Comment-provided video generating apparatus and comment-provided video generating method

ABSTRACT

A comment-provided video generating apparatus includes: a display priority calculating unit which calculates a display priority for each of the comments and subject regions to which the comments are provided, based on comment information indicating comments to be superimposed and displayed on a video and display positions of the comments, and user information, by calculating a higher display priority for one of the comments and the subject regions specified by user information; a comment superimposing method determining unit which determines, as a method for superimposing the comments onto a video, an order of displaying the comments and the subject regions such that a comment or a subject region having a higher display priority is displayed at a position frontward than the others, according to the display priority; and an image composing unit which superimposes the comments on the video, according to the comment superimposing method.

CROSS REFERENCE TO RELATED APPLICATIONS

This is a continuation application of PCT International Application No.PCT/JP2013/004150 filed on Jul. 4, 2013, designating the United Statesof America, which is based on and claims priority of Japanese PatentApplication No. 2012-161532 filed on Jul. 20, 2012. The entiredisclosures of the above-identified applications, including thespecifications, drawings and claims are incorporated herein by referencein their entirety.

FIELD

One or more exemplary embodiments disclosed herein relate generally to acomment-provided video generating apparatus and a comment-provided videogenerating method for generating a comment-provided video including acomment related to a subject.

BACKGROUND

Recent advancement in network and diffusion of mobile terminals havemade it easy and common for users to enjoy communicating video contentor still image content with partners at remote places. For example,there is a service for enabling users to write sentences indicatingtheir tastes, ideas, or the like at any time points on the time axis ofan already provided content item. Such communication using the contentitem is expected to be on the increase with development of displaydevices and communication techniques.

In order to realize such communication, Patent Literature 1 shows atechnique for superimposing a comment region including a comment onto avideo and distributing the video through the Internet.

CITATION LIST Patent Literature

-   [PTL 1]-   Japanese Unexamined Patent Application Publication No. 2008-148071-   [PTL 2]-   Japanese Unexamined Patent Application Publication No. 2011-61528-   [PTL 3]-   Japanese Unexamined Patent Application Publication No, 2004-128614-   [PTL 4]-   International Patent Application Publication No. 2010/116820-   [PTL 5]-   Japanese Unexamined Patent Application Publication No. 2009-81592-   [PTL 6]-   Japanese Patent Publication No. 4994525

Non Patent Literature

-   [NPL 1]-   Konpyuta Bijon—Gijutsu Hyoron to Shorai Tembo (Computer    Vision—Technical Review and Foresight), edited by Takashi Matsuyama    et. al., Dai 3 Sho “Kamera Kyariburasion” (Chapter 3 “Camera    Calibration”, published in 1998, by Shin Gijutsu Komunikasionzu    (current name: Advanced Communication Media co.), ISBN: 4915851176

SUMMARY Technical Problem

However, according to a conventional comment superimposing method, inthe case where comment regions are overlapped with each other in acontent, a comment that a user wants to read may be occluded by acomment region of another comment, or a comment region may occlude asubject that the user wants to see in the content, resulting in adecrease in visibility of the comment or the subject that the user wantsto read or see without occlusion.

One non-limiting and exemplary embodiment provides a comment-providedvideo generating apparatus and a comment-provided video generatingmethod for generating a video (comment-provided video) in which acomment or a subject which is important to a user is displayed with anincreased visibility in a userfriendly manner.

Solution to Problem

In one general aspect, the techniques disclosed here feature acomment-provided video generating apparatus including: a data obtainingunit configured to obtain a video from a video server, and obtain, froma comment information storage server, comment information including (i)comments regarding the video, and (ii) display time and displaypositions of comment regions including the comments in the video; aninput receiving unit configured to receive, as a user input, userinformation including a specified time and a specified position in thevideo; a display priority calculating unit configured to calculate adisplay priority for each of the comment regions and subject regions towhich the comments are provided, based on the user information andassociation information indicating associations between the comments andthe subject regions, by calculating higher display priorities for one ofthe comment regions and the subject regions which corresponds to thespecified time and the specified position included in the userinformation, and for either (i) one of the subject regions which isprovided with a comment included in the comment region which correspondsto the specified time and the specified position included in the userinformation, or (ii) one of the comment regions which includes a commentprovided to the subject region which corresponds to the specified timeand the specified position included in the user information, than theother display priority for each of the other subject regions and theother comment regions; a comment superimposing method determining unitconfigured to determine a superimposing method for superimposing acomment region or a subject region which has a higher display priorityat a position more frontward than a position of a comment region or asubject region which has a lower display priority according to thedisplay priority calculated by the display priority calculating unitwhen the comment regions and the subject regions are displayed in thevideo on an image display unit; an image composing unit configured togenerate a comment-provided video by composing the comment regions inthe video, according to the superimposing method determined by thecomment superimposing method determining unit; and an image output unitconfigured to output the comment-provided video generated by the imagecomposing unit, wherein the image output unit is configured to output,onto the image display unit, the comment-provided video in which either(i) a first pair of the one of the subject regions and a correspondingone of the comment regions which includes a comment provided to thesubject region or (ii) a second pair of the one of the comment regionsand a corresponding one of the subject regions to which a commentincluded in the comment region is provided is superimposed and displayedon the other subject regions and the corresponding comment regionsincluding comments provided to the other subject regions, the subjectregion in the first pair and the comment region in the second paircorresponding to a user-selected region located at the specifiedposition indicated by the user information.

General and specific aspect(s) disclosed above may be implemented usinga system, a method, an integrated circuit, a computer program, or acomputer-readable recording medium such as a CD-ROM, or any combinationof systems, methods, integrated circuits, computer programs, orcomputer-readable recording media.

Additional benefits and advantages of the disclosed embodiment will beapparent from the Specification and Drawings. The benefits and/oradvantages may be individually obtained by the exemplary non-limitingembodiment and features of the Specification and Drawings, which neednot all be provided in order to obtain one or more of such benefitsand/or advantages.

Advantageous Effects

According to one or more exemplary embodiments or features disclosedherein make it possible to generate a video in which a comment regionincluding a comment is superimposed with an increased visibility.

BRIEF DESCRIPTION OF DRAWINGS

These and other advantages and features will become apparent from thefollowing description thereof taken in conjunction with the accompanyingDrawings, by way of non-limiting examples of embodiments disclosedherein.

FIG. 1 is a diagram showing an example of a structure of a videodistributing system in Embodiment 1.

FIG. 2A is a diagram showing a time flow of provision of comments to avideo.

FIG. 2B is a diagram showing relationships between dates and time in theflow and the comments to be displayed.

FIG. 3 is a diagram showing an entire configuration of the videodistributing system including a comment-provided video generatingapparatus.

FIG. 4 is a diagram showing another entire configuration of the videodistributing system including the comment-provided video generatingapparatus.

FIG. 5A is a diagram showing an example of data to be stored in acomment information storage server.

FIG. 5B is a diagram showing an example of data to be stored in a userID information server.

FIG. 5C is a diagram showing an example of data to be stored in asubject region information server.

FIG. 5D is a diagram showing an example of data to be stored in a videoserver.

FIG. 6 is a block diagram showing a structure of the comment-providedvideo generating apparatus according to Embodiment 1.

FIG. 7A is a diagram showing a hardware structure of thecomment-provided video generating apparatus configured with a computer.

FIG. 7B is a diagram showing the hardware structure of thecomment-provided video generating apparatus configured with thecomputer.

FIG. 8 is a flowchart showing a procedure of operations performed by thecomment-provided video generating apparatus.

FIG. 9A is a diagram showing an example of a video.

FIG. 9B is a diagram showing examples of subject regions in the video.

FIG. 10A is a diagram schematically showing subject region informationobtained by a data obtaining unit.

FIG. 10B is a diagram showing an example of a data structure of thesubject region information.

FIG. 11A is a diagram schematically showing comment regions in commentinformation obtained by the data obtaining unit.

FIG. 11B is a diagram showing an example of a data structure of commentinformation.

FIG. 12 is a diagram showing how comments are input by different usersalong time.

FIG. 13 is a diagram showing an example of user information that isreceived by a user input receiving unit by an input on a touch panel.

FIG. 14 is a diagram for explaining the concept of superimposition ofcomment regions and subject regions.

FIG. 15A is a diagram showing a display example of a video includingframes in which comment regions are displayed to track subject regions.

FIG. 15B is a diagram showing a display example of the video includingthe frames in one of which one of the comment regions is temporarilystopped without tracking a corresponding one of the subject regions.

FIG. 16A is a diagram showing a display example of the video includingthe frames in two of which the two comment regions are integrated intoone when displayed.

FIG. 16B is a diagram showing a display example of the video includingthe frames in two of which the two comment regions are integrated intoone when displayed.

FIG. 17A is a diagram showing a display example of the video includingthe frames in which both of the comment regions are displayed.

FIG. 17B is a diagram showing a display example of the video includingthe frames in one of which one of the comment regions is temporarilyerased or made into a non-display state when the other comment region isdisplayed.

FIG. 18A is a diagram showing a display example of the video includingthe frames in one of which occluding one of the comment regions isdisplayed transparent.

FIG. 18B is a diagram showing a display example of the video includingthe frames in one of which occluded one of the comment regions isdisplayed transparent.

FIG. 19 is a diagram for explaining relationships between userspecifications and display priorities.

FIG. 20 is a flowchart of processes executed by a display prioritycalculating unit.

FIG. 21A is a diagram showing subject regions shown by subject regioninformation in a general video.

FIG. 21B is a diagram showing a display example of a generalcomment-provided video.

FIG. 21C is a diagram showing a display example of the video when one ofthe subject regions is selected.

FIG. 21D is a diagram showing a display example of the video when one ofthe comment regions is selected.

FIG. 21E is a diagram showing a display example of the video when a pairof a subject region and a comment region among the subject regions andthe comment regions is selected.

FIG. 21F is a diagram showing a display example of the video when all ofthe subject regions are selected without selecting the background.

FIG. 22 is a block diagram showing a structure of a comment-providedvideo generating apparatus including a user information obtaining unit.

FIG. 23 is a diagram showing a change in a superimposition order ofcomment regions and subject regions, made together with an activity of auser on the Internet.

FIG. 24 is a diagram showing a superimposition order of comment regionsand subject regions and a viewing history of a video, both of which areupdated each time the video is viewed.

FIG. 25A is a diagram showing a concept of a comment that follows amotion generated by a comment-provided video generating apparatusaccording to Embodiment 1 or Embodiment 2.

FIG. 25B is a diagram showing a concept of the comment that follows themotion generated by the comment-provided video generating apparatusaccording to Embodiment 1 or Embodiment 2.

FIG. 26 shows how comment regions are displayed in the cases in each ofwhich one viewing method that is fast forward, preview, rewind, skip orthe like is performed.

FIG. 27 is a diagram showing an example of a structure of a videodistributing system in Embodiment 2.

FIG. 28 is a block diagram showing a structure of the comment-providedvideo generating apparatus according to Embodiment 2.

FIG. 29A is a block diagram showing a configuration of a systemincluding an input device and an input supporting apparatus.

FIG. 29B is a flowchart of processes executed by the input supportingapparatus.

FIG. 30A is a diagram showing procedures for transitioning from a videoviewing state into a comment input state.

FIG. 30B is a diagram showing image screen display examples in a commentinput state.

FIG. 31 is a block diagram showing a structure of a comment-providedvideo generating apparatus according to Embodiment 3.

FIG. 32 is a diagram showing an example of a structure of a videodistributing system in Embodiment 3.

FIG. 33 is a diagram showing an example of a usage scene of thecomment-provided video generating apparatus according to Embodiment 3.

FIG. 34 is a diagram showing positional relationships between subjectsand a user of the comment-provided video generating apparatus accordingto Embodiment 3.

DESCRIPTION OF EMBODIMENTS

(Underlying Knowledge Forming Basis of the Present Disclosure)

In relation to conventional comment superimposing methods disclosed inthe Background section, the inventors have found the problems below.

Patent Literature 1 discloses a system including a video server and acomment distribution server in which comments written by users throughnetworks are stored firstly, and then are superimposed on videos anddistributed. Each comment newly written in a video by a user when thevideo is distributed and reproduced is managed in the commentdistribution server in a form according to a video time axis in additionto a real time point at which the comment is provided by the user. Thecomment written in the video is, for example, distributed along thevideo time axis irrespective of user comment time (a time point at whichthe user provided the comment) in the future distribution. In many casesof video distribution services assumed in Patent Literature 1, severalcomments are provided to a particular scene in the same frame of avideo, and comment regions including the comments are dominant in a partof the particular scene. In order to prevent such an overlap betweencomment regions, some schemes have been disclosed which include toadjust speeds of comment regions moving in a video on a display screenand to adjust the display position of an old comment if a new comment isto be provided at the display position of the old comment.

Patent Literature 2 shows a system for allowing users to submitmanuscript comments, while Patent Literature 1 handles characterinformation as comments. The system in Patent Literature 2 is designedfor use in the case where a plurality of comment regions each includinga manuscript comment in a target video are in contact with each other ora partial video region in a target video overlaps a manuscript region.In such a case, the system selects and displays an image effectrepresenting an overlap between the comments or the comment and thepartial video region based on a user instruction. Here, the partialvideo region is a region discriminated based on brightness, color,contrast, and so on.

On the other hand, Patent Literature 3 discloses a chat system capableof composing and displaying character data in balloons in a video. Thechat system in Patent Literature 3 determines a method for displayingeach of balloons which appears in a video during a chat at a displayposition other than positions of previously provided balloons and userface images arranged and composed at predefined positions on abackground image in the video.

In addition, there has been proposed an approach for adding characterinformation to videos without use of the Internet.

Patent Literature 4 shows a video recorder which displays supplementaldata to captured videos. Patent Literature 4 proposes an approach for,for example, displaying supplemental data using characters and aballoon, and changing the display position of the balloon according to amotion of an object in the video.

In addition, Patent Literature 5 shows an approach for inputtingcharacter information and illustration information to content items heldpersonally using mobile terminals. Patent Literature 5 proposes afunction for providing a visual effect of causing the characterinformation and illustration information to track an object in an objectframe manually specified by a user, using a motion vector defined by theMoving Picture Experts Group (MPEG) in the frame.

In view of the conventionally proposed approaches for inserting commentsinto videos as described above, one or more embodiments disclosed herehave been conceived particularly focusing on the fact that most ofvideos distributed by content providers and content items heldpersonally (hereinafter referred to as “actual image content items”)includes a plurality of objects (subjects) per scene. Conventionally, inthe case of communication through comments on such an actual imagecontent item, it is difficult to generate a comment-provided video inwhich a comment or a subject important to a user is not occluded,especially when a plurality of comment regions and a plurality ofobjects are present therein at the same time.

Patent Literature 1 discloses the approach for preventing an overlapbetween comment regions, but does not solve the problem that the videoin which the comment regions are superimposed is very difficult to see.The approach in Patent Literature 1 is for displaying the commentregions without considering the positions of the objects in the video.Thus, in a particular case where one of the objects in the video is anobject that the user wants to see without occlusion, the visibility ofthe object inevitably decreases.

The system in Patent Literature 2 regards, to be the object, the partialvideo region having a motion relatively different from the motion of thebackground in the video, and determines that the object affects one ormore of the comment regions. However, Patent Literature 2 does notspecifically disclose an approach for generating the partial videoregion in the case where a plurality of objects are present. In thiscase, the visibility of the object may be maintained when one of thecomment regions and one of the objects becomes in contact with eachother and the comment region “adheres to” the outer periphery of theobject. However, since the relationships with the other objects are notconsidered, the visibility of the other objects inevitably decreases.

The approach disclosed in Patent Literature 3 is for arranging aplurality of balloons for a plurality of objects representing chatmembers, on the premise that these objects are composed at predeterminedpositions. Thus, this approach cannot be used for actual image contentitems.

Patent Literatures 4 and 5 disclose display of balloons which trackobjects or balloons with consideration of motions of the objects, but donot at all disclose a method for displaying balloons (comment regions)with consideration of region information of a plurality of objects.

In one general aspect, the techniques disclosed here feature acomment-provided video generating apparatus including: a data obtainingunit configured to obtain a video from a video server, and obtain, froma comment information storage server, comment information including (i)comments regarding the video, and (ii) display time and displaypositions of comment regions including the comments in the video; aninput receiving unit configured to receive, as a user input, userinformation including a specified time and a specified position in thevideo; a display priority calculating unit configured to calculate adisplay priority for each of the comment regions and subject regions towhich the comments are provided, based on the user information andassociation information indicating associations between the comments andthe subject regions, by calculating a highest display priority for oneof the comment regions and the subject regions which corresponds to thespecified time and the specified position included in the userinformation, and calculating a next highest display priority for either(i) one of the subject regions which is provided with a comment includedin the comment region which corresponds to the specified time and thespecified position included in the user information, or (ii) one of thecomment regions which includes a comment provided to the subject regionwhich corresponds to the specified time and the specified positionincluded in the user information; a comment superimposing methoddetermining unit configured to determine a superimposing method forsuperimposing a comment region or a subject region which has a higherdisplay priority at a position more frontward than a position of acomment region or a subject region which has a lower display priorityaccording to the display priority calculated by the display prioritycalculating unit when the comment regions and the subject regions aredisplayed in the video on an image display unit; an image composing unitconfigured to generate a comment-provided video by composing the commentregions in the video, according to the superimposing method determinedby the comment superimposing method determining unit; and an imageoutput unit configured to output the comment-provided video generated bythe image composing unit, wherein the image output unit is configured tooutput, onto the image display unit, the comment-provided video in whicheither (i) a first pair of the one of the subject regions and acorresponding one of the comment regions which includes a commentprovided to the subject region or (ii) a second pair of the one of thecomment regions and a corresponding one of the subject regions to whicha comment included in the comment region is provided is superimposed anddisplayed on the other subject regions and the corresponding commentregions including comments provided to the other subject regions, thesubject region in the first pair and the comment region in the secondpair corresponding to a user-selected region located at the specifiedposition indicated by the user information.

With this structure, it is possible to calculate the higher displaypriority for the one of the comment regions and subject regions whichcorresponds to the specified time and the specified position(coordinates) included in the user information and specified by the userwho is browsing the video, compared to the display priorities for theother regions, and based on the calculated display priority, to generatethe comment-provided video by superimposing and composing the commentregions of the comments in the video (video data). Here, it is possibleto realize the userfriendly display while maintaining the visibility ofthe comment or subject region which is important to the user bysuperimposing and displaying some of the comment regions on particularones of the subject regions and comment regions and not superimposingand displaying any one of the comment regions on the comment or subjectregion which is important to the user and thus is specified by the inputfrom the user (in the latter case, the non-specified regions may bedisplayed behind the specified comment or subject region). In this way,it is possible to generate the comment-provided video in which thecomment or subject that the user wants to see without occlusion has anincreased visibility among the other comment and subject regionssuperimposed and displayed therein.

In addition, information indicating associations between the commentregions and the subject regions is stored in advance, and thus it ispossible to generate the comment-provided video in which one of thesubject region and the comment region which corresponds to theuser-specified (user-input) time or position (coordinates) in the videoincluded in the user information (user input) is superimposed at theposition more frontward than the position of the other regions whendisplayed therein.

For example, among the comment regions other than (i) the comment regioncorresponding to the specified time and the specified position includedin the user information or (ii) the comment region including the commentprovided to the subject region corresponding to the specified time andthe specified position, the comment superimposing method determiningunit may be further configured to determine, based on depth relationshipbetween the subject regions in a frame of the video, a superimposingmethod for superimposing a comment region including a comment providedto a subject region located frontward on the other subject regions andthe corresponding comment regions located backward, and when the commentregions other than (i) the comment region corresponding to the specifiedtime and the specified position or (ii) the comment region including thecomment provided to the subject region corresponding to the specifiedtime and the specified position may be displayed in the video on theimage display unit, the comment region including the comment provided tothe subject region located frontward is superimposed on the othersubject regions and the corresponding comment regions located, based onthe depth relationship between the subject regions.

With this structure, it is possible to provide the comment-providedvideo in which the depth relationships between the subject regions otherthan the subject region specified by the user are reflected in the depthrelationships between the corresponding comment regions.

In addition, the display priority calculating unit may be configured tocalculate a higher display priority for a comment region including acomment which is provided later in either the display time included inthe comment information or in real submission time of the comments, thecomment superimposing method determining unit may be further configuredto determine, based on the display priority, a superimposing method forsuperimposing the comment region including the comment provided later ineither the display time included in the comment information or in thereal submission time of the comments at a more frontward position in thevideo on the image display unit, and the comment region including thecomment provided later in either the display time included in thecomment information or in the real submission time of the comments maybe superimposed and displayed at the more frontward position in thevideo on the image display unit.

With this structure, it is possible to provide the comment-providedvideo in which the comment region provided later in either the displaytime or the submission time among the comment regions other than theuser-specified one is superimposed and displayed at the more frontwardposition.

In addition, the above-described comment-provided video generatingapparatus may further include a user information obtaining unitconfigured to obtain the user information from a user ID informationserver, wherein, when user ID information in the user ID informationserver is updated while the user is viewing the video, the displaypriority calculating unit may be further configured to calculate a newdisplay priority for each of the comment regions and the subjectregions, based on the updated user ID information.

With this structure, it is possible to provide the comment-providedvideo in which a change in the user's tastes is reflected without anydirect user specification on the video.

In addition, the comment superimposing method determining unit may befurther configured to: calculate, for each of pixels in an overlap areabetween two regions which are either (i) one of the subject regions andone of the comment regions or (ii) two of the comment regions, atransparency of the pixel to be displayed semi-transparent, based on adisplay priority ratio of the display priorities calculated for the tworegions including the pixels by the display priority calculating unit;and determine a superimposing method for displaying more opaquely one ofthe two regions which is superimposed at the position more frontwardthan the position of the other one of the two regions according to thecalculated transparency.

With this structure, it is possible to provide a display effect ofdisplaying transparent both the pixels in one of the comment regions andeither the pixels in another one of the comment regions or in one of thesubject regions.

In addition, the comment superimposing method determining unit may befurther configured to determine, based on information describing anocclusion relationship between subject regions among the subjectregions, a superimposing method for displaying comment regions includingcomments corresponding to the subject regions in the occlusionrelationship by temporally stopping the comment regions at positionsbefore a start of an occlusion period in which the subject regions arein the occlusion relationship, and causing the comment regions tore-start tracking the subject regions after the occlusion period, and inthe comment-provided video displayed on the image display unit, thecomment regions including the comments corresponding to the subjectregions in the occlusion relationship may be temporally stopped at thepositions before the start of the occlusion period in which the subjectregions are in the occlusion relationship, and may be caused to re-starttracking the subject regions after the occlusion period.

With this structure, it is possible to provide the comment-providedvideo in which both of the comment regions of the comments which areprovided to the subject regions in the occlusion relationship and whichthe user wants to see are displayed in the userfriendly manner withoutocclusion.

In addition, the comment superimposing method determining unit may befurther configured to determine, based on information describing anocclusion relationship between subject regions among the subjectregions, a superimposing method for displaying comment regions includingcomments provided to the subject regions in the occlusion relationshipby integrating the comment regions which are balloons at positionsbefore a start of an occlusion period in which the subject regions arein the occlusion relationship into a comment region which is a balloon,dividing the comment region which is the balloon into separate commentregions which are balloons after the occlusion period, and causing thecomment regions to track the subject regions, and in thecomment-provided video displayed on the image display unit, the commentregions which are the balloons at the positions before the start of theocclusion period in which the subject regions are in the occlusionrelationship may be integrated into the comment region which is theballoon, the comment region which is the balloon may be divided into theseparate comment regions which are the balloons after the occlusionperiod, and the comment regions may be caused to track the subjectregions.

With this structure, it is possible to provide the comment-providedvideo in which the subject regions in the temporary occlusionrelationship and the comments provided thereto are emphasized anddisplayed in the userfriendly manner.

In addition, the comment superimposing method determining unit may befurther configured to determine, based on information describing anocclusion relationship between subject regions among the subjectregions, a superimposing method for displaying only a comment regioncorresponding to an occluding subject region closer to a location of acamera used to capture the video among comment regions includingcomments provided to the subject regions in the occlusion relationshipduring an occlusion period in which the subject regions are in theocclusion relationship, and displaying all of the comment regionsincluding comments provided to the subject regions after the occlusionperiod, and in the comment-provided video displayed on the image displayunit, only the comment region corresponding to the occluding subjectregion closer to the location of the camera used to capture the videoamong the comment regions including the comments provided to the subjectregions in the occlusion relationship may be displayed during theocclusion period in which the subject regions are in the occlusionrelationship, and all of the comment regions including the commentsprovided to the subject regions may be displayed after the occlusionperiod.

With this structure, it is possible to provide the comment-providedvideo in which relationships between the subjects in real scenes areemphasized and displayed in the userfriendly manner.

In addition, the comment superimposing method determining unit may beconfigured to determine a superimposing method and a transparency forsuperimposing the comment regions by displaying one of the commentregions which includes the comment provided to the occluding subjectregion semi-transparent, and all of the comment regionsnon-semi-transparent after the occlusion period, and in thecomment-provided video displayed on the image display unit, the commentregion including the comment provided to the occluding subject regionmay be displayed semi-transparent, and all of the comment regions may bedisplayed non-semi-transparent after the occlusion period.

With this structure, it is possible to display the comment-providedvideo in which the subject regions in the temporary occlusionrelationship and the comments provided thereto are emphasized anddisplayed in the userfriendly manner.

In addition, the comment superimposing method determining unit may beconfigured to further determine a superimposing method for displayingonly a comment region which is displayed at a foremost position in eachof frames of the video in a case of normal reproduction among thecomment regions in the frame, during a period in which a viewing methodwhich is other than the normal reproduction and includes fast forward,preview, rewind, and skip is performed on the comment-provided video,and in the comment-provided video displayed on the image display unit,only the comment region which is displayed at the foremost position ineach of the frames of the comment-provided video in the case of thenormal reproduction among the comment regions in the frame may bedisplayed, during the period in which the viewing method which is otherthan the normal reproduction and includes fast forward, preview, rewind,and skip may be performed on the comment-provided video.

With this structure, by displaying only such important comments when theuser needs to understand information in a frame in a short time durationin the case of fast forward, rewind, or the like, it is possible toprovide the comment-provided video which reflects the user's tastes andthus can be read and understood by the user in such short time duration.

In addition, the user information obtaining unit may be furtherconfigured to obtain viewing history of the user stored externally or inthe user ID information server, when the comment-provided video whichhas been viewed by a user may be viewed by the user, the displaypriority calculating unit may be further configured to provide a commentregion or a subject region displayed at a foremost position when viewedlast time by the user with a lowest display priority, and thus, in thecomment-provided video displayed on the image display unit, the commentregion displayed at the foremost position when viewed last time by theuser may be displayed at a rearmost position.

With this structure, the depth relationships between the comment regionsare updated each time the user watches the video. Thus, it is possibleto provide the same user with the video having the same contentpresented in a different superimposition and display manner each timethe user watches the video.

In addition, the display priority calculating unit may calculate, foreach of the plurality of comments, association information indicating anassociation between the comment and the subject region to which thecomment is provided, based on the display position of the commentindicated by the comment information, and, based on the calculatedassociation information, calculate the display priority of each of thecomment regions and the subject regions by calculating a highest displaypriority for one of the comment regions and the subject regions whichcorresponds to the specified time and the specified position included inthe user information, and a next highest display priority for either (i)one of the subject regions which is provided with a commentcorresponding to the specified time and the specified position includedin the user information, or (ii) one of the comment regions whichincludes a comment provided to the subject region which corresponds tothe specified time and the specified position included in the userinformation.

With this structure, since the association information between thecomment region and the subject region is calculated, even when no suchassociation information is obtained in advance, it is possible togenerate the comment-provided video in which one of the subject regionand the comment region which corresponds to the user-specified(user-input) time or position (coordinates) in the video included in theuser information (user input) is superimposed at the position morefrontward than the position of the other regions when displayed therein.

In addition, the comment information may further include, for each ofthe plurality of comments, information indicating a display position anda display shape of a balloon in which the comment is displayed. Based onthe information indicating the display position and the display shape ofthe balloon, the display priority calculating unit may calculate, foreach comment, association information indicating the subject region towhich the comment is provided, and based on the calculated associationinformation, may calculate by calculating a highest display priority forone of the comment regions and the subject regions which corresponds tothe specified time and the specified position included in the userinformation, and a next highest display priority for either (i) one ofthe subject regions which is provided with a comment corresponding tothe specified time and the specified position included in the userinformation, or (ii) one of the comment regions which includes a commentprovided to the subject region which corresponds to the specified timeand the specified position included in the user information.

With this structure, when the subject region is specified by the user,it is possible to provide the comment-provided video in which thespecified subject region and the comment region of the comment providedto the subject region are superimposed on the other subject regions andcomment regions when displayed therein.

In addition, in the case where the comment regions (comments)corresponding to the specified time and the specified position includedin the user information exists, the display priority calculating unitmay further calculate the display priority of each of the commentregions and the subject regions by calculating, for each of the commentregions other than the comment region, a display priority lower than thedisplay priorities for all of the subject regions. In the opposite casewhere no such comment region corresponding to the specified time and thespecified position included in the user information exists, the displaypriority calculating unit may further calculate the display priority foreach of the comment regions and the subject regions by calculating, forall of the comment regions, display priorities lower than the displaypriorities for all of the subject regions.

With this structure, it is possible to superimpose and display thesubjects on the comment regions other than the comment region specifiedby the user.

In addition, in the case where the subject region corresponding to thespecified time and the specified position included in the userinformation exists, the display priority calculating unit may furthercalculate the display priority for each of the comment regions and thesubject regions by calculating, for the subject regions other than thesubject region, display priorities lower than the display priorities forall of the comment regions. In the opposite case where no such commentcorresponding to the specified time and the specified position includedin the user information exists, the display priority calculating unitmay further calculate the display priority for each of the commentregions and the subject regions by calculating, for all of the subjectregions, display priorities lower than the display priorities for all ofthe comment regions.

With this structure, it is possible to provide the comment-providedvideo in which the subject region specified by the user and another oneof the subject regions can be displayed with one of the commentsinterposed therebetween.

Alternatively, the display priority calculating unit may calculate ahigher display priority for each of the plurality of comments includingthe specified comment corresponding to the specified time and thespecified position included in the user information and other commentsprovided by a provider of the specified comment, compared to the othercomments provided by other users.

With this structure, it is possible to superimpose and display thecomments provided by the provider of the comment specified by the userat the position more frontward than the positions of the other commentswithout requiring the user to specify the comments of the provider onthe display screen one by one.

In addition, the comment-provided video generating apparatus may includea subject region information generating unit which generates subjectregion information indicating subject regions generated in a frame ofthe video by dividing the frame.

In general, a video is not provided in advance any information regardingsubject regions included therein. However, with this structure, evenwhen subject regions are unknown, it is possible to detect the subjectregion information by the division, and utilizes the detected subjectregion information.

In addition, when the user adds a new comment to the video, it is alsogood to estimate one of the subject regions and the comment regionswhich is highly likely to be provided with the comment by the user, andmodify display of the video by superimposing and displaying theestimated region at a frontward position. The comment-provided videogenerating apparatus having this structure can, as the input supportingapparatus supports, provide the comment-provided video in which theregion that is highly likely to be provided with the comment by the usercan be easily specified by the user.

In one general aspect, the techniques disclosed here feature acomment-provided video generating apparatus according to an aspectincludes: a subject region information generating unit configured to:obtain a video captured by a camera from the mobile terminal includingthe camera used to capture an image of a user positioned in front of thecamera, a display which is provided at a position that allows the userto visually recognize the display, a Global Positioning System (GPS),and a compass; and generate subject region information indicating aplurality of subject regions in a frame of the obtained video based oncontent of the video, the subject regions being generated in the frameby dividing the frame; a data obtaining unit configured to obtaincomment information including (i) GPS information of other mobileterminals located in a predetermined range from the user and (ii) aplurality of comments associated one to one to the other mobileterminals; an input receiving unit configured to receive, as a userinput, user information including a specified position in the video; acomment position estimating unit configured to calculate positions ofthe other mobile terminals in the video, based on the GPS information ofthe mobile terminal of the user and the other mobile terminals andinformation of the compass of the mobile terminal, and associate one toone subject regions presenting subjects who are users of the othermobile terminals and the comments included in the comment informationobtained by the data obtaining unit, based on the calculated positionsof the other mobile terminals; a display priority calculating unitconfigured to calculate, based on the comment information and the userinformation, a display priority for each of the comment regions whichare superimposed and displayed in the video and the subject regions inthe video by calculating, for a comment region including a commentcorresponding to a subject region located at a display positionspecified by the user information among the comment regions, a displaypriority which is higher than the other display priority for each of theother subject regions and the other comment regions; a commentsuperimposing method determining unit configured to determine, as asuperimposing method for displaying the plurality of comment regions inthe video, one of first and second methods according the displaypriority calculated by the display priority calculating unit: the firstmethod being for determining a display order for displaying the commentregions and the subject regions in the video on an image display unit bysuperimposing a comment region or a subject region which has a higherdisplay priority on a comment region or a subject region which has alower display priority according to the display priority calculated bythe display priority calculating unit; and the second method being fordetermining a transparency for displaying a comment region or a subjectregion which has a higher transparency more opaquely according to thedisplay priority; an image composing unit configured to superimpose thecomment regions in the video, according to the comment superimposingmethod determined by the comment superimposing method determining unit;and an image output unit configured to output the comment-provided videoin which the comment regions are superimposed by the image composingunit.

With this structure, it is possible to generate and display acomment-provided video in which a comment region corresponding to auser-specified subject region is not occluded by another comment regionwhen a comment corresponding to a subject in a video captured by theuser is displayed as in the case of using an Augmented Reality (AR)technique.

These general and specific aspects may be implemented using a system, amethod, an integrated circuit, a computer program, or acomputer-readable recording medium such as a CD-ROM, or any combinationof systems, methods, integrated circuits, computer programs, orcomputer-readable recording media.

Hereinafter, exemplary non-limiting embodiments are described in detailwith reference to the accompanying Drawings. The exemplary embodimentdescribed below shows a general or specific example. The numericalvalues, shapes, materials, structural elements, the arrangement andconnection of the structural elements, steps, the processing order ofthe steps etc. shown in the following exemplary embodiments are mereexamples, and therefore do not limit the scope of the appended Claimsand their equivalents. Therefore, among the structural elements in thefollowing exemplary embodiments, structural elements not recited in anyone of the independent claims are described as arbitrary structuralelements.

Embodiment 1

FIG. 1 shows a simple example of a configuration of a video distributingsystem according to this embodiment. As in Patent Literature 1, thevideo distributing system includes a video server, a comment informationstorage server, a user ID information server, and a region informationserver. Each server may be managed on the same personal computer (PC) oron different PCs.

Users A to D shown in FIG. 1 browse a video distributed to theirterminals (such as smart phones, PCs, and tablet PCs) through networks.In this example, the users A and B provide comments to the video viakeyboards, software keyboards, or the like of their terminals. Eachcomment is character information, and is provided to the video, inassociation with a user-specified temporal position (corresponding to aspecified time or a specified frame number) and a spatial position(coordinates) in the video. The term “video” used in the followingdescriptions refers to as video or video data.

Each of the comments provided by the users is stored into the commentinformation storage server at any time, in association with an image towhich the comment is provided and various kinds of information such as auser ID of the user, a time point and a set of coordinates of thecomment (in the video), and real time at which the comment is submitted.In the case where, other users C and D view the video after the users Aand B provide the comments onto the video, the comments of the users Aand B are distributed from the comment information storage servertogether with other comments associated with the video. In addition, thecomments are composed on the video, based on the information (video timethat is time elapsed in the video, coordinates, etc.) associated withthe comments. In this way, the video on which the comments aresuperimposed is displayed is provided. How the comments are displayed inthe above processes are closely related to real time (comment submissiontime) at which the comments are submitted and comment display time inthe video.

FIG. 2A shows relationships between the comments to be displayed in avideo and time in the video. Taking the example in which the users A andB provide the comments to the video, a flow of providing and displayingcomments is described in detail below. Each comment may have a lengthspecified by the user, or a predetermined length (for example, 2seconds, or the like). Here, “second” is used below as a unit time, butthe second may be replaced with the number of frames.

As shown in FIG. 2A, the comment of the user A is displayed from SAsecond to EA second in the video time, and the comment of the user B isdisplayed from SB second to EB second in the video time. Accordingly,when the video is viewed, only the comment of the user A is displayedfrom SA second to SB second, both of the comments of the users A and Bare displayed from SB second to EA second, and only the comment of theuser B is displayed from EA second to EB second.

In addition, FIG. 2B shows relationships between dates and time in areal time flow and comments to be displayed in the video. The user D whoviews the video between the comment provision (writing) by the user Aand the comment provision (writing) by the user B can view only thecomment of the user A. On the other hand, the user C who views the videoafter the comment provision by the users A and B can view both of thecomments of the users A and B.

In many cases, a comment region including a comment provided later inthe video time or the real time is generally superimposed and composedon a comment region including a comment provided before thelater-provided comment, so as to increase the visibility of thelater-provided comment.

FIG. 3 is a diagram showing a simple configuration of a videodistributing system including the comment-provided video generatingapparatus. The comment-provided video generating apparatus 100distributes a comment-provided video to terminals of individuals throughnetworks. In addition, the comment-provided video generating apparatus100 obtains comments to be provided, information of users who input thecomments, etc. from the servers and terminals through networks.

Functions of the comment-provided video generating apparatus 100 may beincluded in each of the terminals of the individuals. FIG. 4 shows suchan example. In this structure, the comment-provided video generatingapparatus 100 obtains information from each server via a network, andperforms processes in the terminal, and displays a comment-providedvideo using the terminal.

FIG. 5A to 5D show examples of data items to be stored in a commentinformation storage server, a user ID information server, a subjectregion information server, and a video server, respectively. Each serverstores information used for associating each comment and a correspondingvideo and subject region. In this embodiment, these information itemsare used to determine a comment superimposing method. The informationand comment superimposing method are described in detail later.

FIG. 6 shows a structure of the comment-provided video generatingapparatus 100 in Embodiment 1. The comment-provided video generatingapparatus 100 includes a comment superimposition information generatingunit 101, an image composing unit 102, and an image output unit 103. Thecomment-provided video generating apparatus 100 generates acomment-provided video based on a video 110, subject region information111, comment information 112, and user information 113, and outputs themonto a display 120 (which is an image display unit). The display 120displays the comment-provided video output from the comment-providedvideo generating apparatus 100. The video 110, the subject regioninformation 111, and the comment information 112 are stored in the videoserver, the comment information storage server, and the subject regioninformation server, respectively. The comment superimpositioninformation generating unit 101 obtains these information items throughcommunication paths such as networks. The comment information 112 storedin the comment information storage server is sequentially updated eachtime a user provides a comment, and the details change as time elapses.

The comment superimposition information generating unit 101 determineshow each comment is to be displayed at a point of video time, based onthe video 110, the subject region information 111, the commentinformation 112, and the user information 113. The image composing unit102 generates a comment-provided video by composing the comment onto thevideo, based on the comment display method determined by the commentsuperimposition information generating unit 101. The image output unit103 outputs, onto the display 120, the comment-provided video generatedthrough the composition by the image composing unit 102.

The comment superimposition information generating unit 101 includes adata obtaining unit 104, an input receiving unit 105, a display prioritycalculating unit 106, and a comment superimposing method determiningunit 107.

The data obtaining unit 104 obtains, from the servers, the video 110,the subject region information 111, and the comment information 112. Thedata obtaining unit 104 may be an interface for reading the video 110stored in the video server directly or via a communication path.

Here, the subject region information 111 includes region informationshowing a result of dividing the subjects in a video into subjectregions, and information indicating depth relationships between thesubject regions. In addition, the comment information 112 includescomment region information such as a comment (character information)provided to the video 110, video time, comment coordinates, real time(comment provision dates and time in FIG. 5A) at which the comment isprovided, and a comment shape of the comment.

The input receiving unit 105 receives user information 113 including atime point and coordinates specified by a user input from a terminal ofa user who is browsing a video. The input receiving unit 105 may be aninterface for reading time and coordinates specified in the video 110 bya user input using, for example, a mouse or a touch onto a touch panelof the terminal, as information included in the user information 113,directly or through a communication path.

The display priority calculating unit 106 calculates display prioritiesfor comments and subjects in the video 110, based on the informationobtained from the data obtaining unit 104 and the input receiving unit105. The display priority calculating unit 106 generates acomment-provided video in which the comment region or the subject regioncorresponding to the specified time and coordinates included in the userinformation 113 received by the input receiving unit 105 is not occludedby another comment region. For this reason, the display prioritycalculating unit 106 calculates, for the user-selected regioncorresponding to the specified time and the specified coordinates, adisplay priority higher than the display priorities for the otherregions, based on the subject region information 111, the commentinformation 112, and the user information 113 obtained by the dataobtaining unit 104. In addition, the display priority calculating unit106 determines, in advance, a rule for providing a higher displaypriority to a comment region of an earlier one of comments which may bedisplayed at the same time, and calculates display priorities for all ofthe comment regions and subject regions based on the predetermined rule.

The comment superimposing method determining unit 107 determines amethod for superimposing the comment regions in the video 110, based onthe display priorities calculated by the display priority calculatingunit 106. The comment superimposing method determining unit 107determines the method for superimposing the subject regions and commentregions in the video 110, based on the display priorities calculated bythe display priority calculating unit 106. In particular, the commentsuperimposing method determining unit 107 superimposes a region (acomment region or a subject region) having a higher display priority onthe other regions each having a lower display priority, therebypreventing the region having the higher display priority from beingoccluded by any of the other regions.

The image composing unit 102 composes one of the subject regions and oneof the comment regions on the comment-provided video 110, based on thecomment superimposing method determined by the comment superimposingmethod determining unit 107. The image output unit 103 outputs, onto thedisplay 120 etc., the comment-provided video after the composition bythe image composing unit 102.

In this DESCRIPTION, “region extraction” is a concept including both anextraction technique for extracting an image region in which a givenobject exists and a division technique for dividing a picture or aregion into regions irrespective of the kind of the object. Theextraction technique and the division technique are largely common toeach other, and thus these techniques are not distinguished from eachother in this DESCRIPTION.

Each of the structural elements (the comment superimposition informationgenerating unit 101, the image composing unit 102, and the image outputunit 103) of the comment-provided video generating apparatus 100 may berealized as software such as a computer-executable program, or may berealized as hardware such as an electric circuit and an integratedcircuit. Each of FIGS. 7A and 7B is a diagram showing a hardwarestructure of the comment-provided video generating apparatus 100 in thisembodiment, which is configured with a computer. FIG. 7A showsstructural elements for realizing the comment-provided video generatingapparatus 100 in the video distributing system in FIG. 3, and FIG. 7Bshows structural elements for realizing the comment-provided videogenerating apparatus 100 in the video distributing system in FIG. 4.These two examples are different only in the output paths to thedisplay, and thus are collectively described below. The storage device210 stores the video 110, the subject region information 111, and thecomment information 112, and outputs these items to an interface (I/F)201. The computer 200 obtains the video 110, the subject regioninformation 111, and the comment information 112, and generates acomment-provided video by performing the above-described processes forgenerating the comment-provided video using the obtained video 110, thesubject region information 111, and the comment information 112.

A display terminal 220 a or a display 220 b obtains and displays apicture generated by the computer 200. The computer 200 includes aninterface (I/F) 201, a CPU 202, a ROM 203, a RAM 204, an HDD 205, and avideo card 206. A program causing the computer 200 to function is storedin the ROM 203 or the HDD 205 in advance. The program is read out fromthe ROM 203 or the HDD 205 by the CPU 202 which is a processor andexpanded. The CPU 202 executes each of instruction codes in the programexpanded in the RAM 204. The I/F 201 is used when a user input madethrough the storage device 210 according to the execution of the programis taken to the RAM 204. Through the 201, the video 110, the subjectregion information 111, and the comment information 112 are taken to theRAM 204 according to the execution of the program. The video card 206outputs the comment-provided video generated according to the executionof the program. Then, the display 120 displays the comment-providedvideo.

The computer program is not limited to the ROM 203 or the HDD 205 whichis a semiconductor, and may be stored in a CD-ROM or the like. Inaddition, the computer program may be transmitted through a wired orwireless network 211, broadcasting, or the like, and then be taken tothe RAM 204 of the computer.

With reference to FIG. 8, operations performed by the comment-providedvideo generating apparatus 100 according to this embodiment aredescribed below. FIG. 8 is a flowchart of operations performed by thecomment-provided video generating apparatus 100 according to thisembodiment.

In FIG. 8, seven Steps S301 to S307 correspond respectively toprocessing units 101 to 107 in FIG. 6. More specifically, the commentsuperimposition information generating unit 101 executes Step S301 thatis a comment superimposition information generating step, the imagecomposing unit 102 executes Step S302 that is an image composing step,and the image output unit executes Step S303 that is an image outputstep. In addition, the comment superimposition information generatingstep S301 includes four steps of Step S304 that is a data obtainingstep, Step S305 that is an input receiving step, Step S306 that is adisplay priority calculating step, and Step 307 that is a commentsuperimposing method determining step. The data obtaining unit 104executes operations in the data obtaining step S304, the input receivingunit 105 executes the input receiving step S305, the display prioritycalculating unit 106 executes the display priority calculating stepS306, and the comment superimposing method determining unit 107 executesthe comment superimposing method determining step S307.

First, the comment superimposition information generating unit 101executes the comment superimposition information generating step S301.The comment superimposition information generating unit 101 includes thedata obtaining unit 104, the input receiving unit 105, the displaypriority calculating unit 106, and the comment superimposing methoddetermining unit 107. The comment superimposition information generatingstep S301 corresponding to the units includes the four steps of the dataobtaining step S304, the input receiving step S305, the display prioritycalculating step S306, and the comment superimposing method determiningunit S307.

In the data obtaining step S304, the data obtaining unit 104 obtains thevideo 110, the subject region information 111, and the commentinformation 112 from the storage device.

In this embodiment, the video 110 obtained by the data obtaining unit104 is assumed to be a broadcast video or a video captured using a homevideo recorder. The video 110 is stored in the video server as describedabove. The data obtaining unit 104 obtains the video 110 through a wiredor wireless network or broadcasting. FIG. 9A shows a frame in the video110. FIG. 9A shows an exemplary case where the video 110 is amusic-related video content, but the video 110 may be of any other kind.To simplify explanation and illustration, FIG. 9B is used toschematically show the frame in the video 110, in the form of an imagein which subject regions are shown as subject region information. Inaddition, FIG. 10A is a schematic diagram showing the subject regioninformation 111 obtained by the data obtaining unit 104, and FIG. 10B isa diagram showing a data structure of the subject region information111. The subject region information 111 includes region informationshowing a result of dividing the subjects in a video into regions asdescribed above, and also includes depth relationships between thesubject regions. For example, as shown in FIG. 10A, it is assumed that abackground subject region 400 and different human subject regions 401 to403 are obtained as region data corresponding to the video 110. In FIG.10A, examples of “region information” items corresponding to therespective regions are shown in rectangles with arrows respectivelypointing the regions. As shown in the example table in FIG. 10B, thesubject region information specifically includes the information itemsbelow.

(1) index: Subject region number index

(2) position attribute: information indicating depth relationshipsbetween subjects in the same frame (In the example in this embodiment, asmaller value is provided for a region closer to a camera.)

(3) coordinates: coordinate values of a pixel belonging to each index

In addition, FIG. 11A is a schematic diagram showing a comment region inthe comment information 112 obtained by the data obtaining unit 104, andFIG. 11B is a diagram showing an example of a data structure of thecomment information 112. FIG. 11A shows an example in which the commentregions 500 to 502 are displayed in the video 110, based on the commentinformation. The comment information 112 specifically includes thefollowing kinds of information for each video as shown in FIG. 11B.

-   -   Video name (the name of the video 110 as a target of comment        provision)    -   Display time (comment display starting time in the video)    -   Comment (comment character string)    -   Submission time (real date and time of comment submission)    -   User (ID of a user who made the comment)    -   Base coordinates (coordinates of a base position of each comment        region)    -   Comment shape (information of the shape of each comment region)    -   Comment index (comment region number index)

In the example of FIGS. 11A and 11B, a record having a comment index “2”in the comment information 112 shows that the comment region 500 isdisplayed at a base point represented by base coordinates ((x, y)=(200,200)), in addition, the comment shape (normal1) of the record determinesthe shape and size of the comment region 500. In this way, it ispossible to know which one of the regions in the frame in the video 110is occluded by the comment region. Here, it is also good to modify theshape or the font size of the comment according to the number ofcharacters in the comment.

In this embodiment, it is assumed that a display time length of thecomment is determined, in advance, to be 00:00:05 (second), The commentregion 500 whose display time is 00:00:20 is displayed from 00:00:20 to00:00:25 in video time of the video 110. At the video time 00:00:20, thecomment region 501 and the comment region 502 are still displayed, andthus three comments in total are displayed. FIG. 12 shows an examplewhere different users A, B, and C input comments. As video timeadvances, the comment region 502 for displaying the comment input by theuser A, the comment region 501 for displaying the comment input by theuser B, the comment region 500 for displaying the comment input by theuser C are superimposed and displayed on a previously presented one ofthe comment regions in the listed order.

It is to be noted that the subject regions are known. For this reason,for example, a rectangle region is calculated which is for enclosing acomment provided at a given display time. In addition, a relativeposition is calculated which is of the base coordinates on the rectangleregion at the given display time. Furthermore, also in the case of eachof frames after the frame at the given display time, rectangle regionseach for enclosing a comment are calculated, and, for each rectangleregion, base coordinates at the display time which remain located at therelatively same position in the rectangle are calculated. Through thisprocess, it is also good to provide a comment-provided video in whichthe provided comment can be seen tracking the subject region to whichthe comment was provided.

Alternatively, it is also good to store, in advance, comment informationincluding different display time lengths for the respective comments,and display these comments for the display time lengths. When aplurality of comment regions are displayed in either case, display ofone or more of the subject regions 401 to 403 is hampered, resulting ina decrease in the visibility as shown in FIG. 11A

With reference to FIG. 8 again, in Step S305, the input receiving unit105 receives user information 113 that the user inputs, when viewing avideo, to select a region to be specified in the video using a mouse, auser interface (UI) for receiving input on a touch panel. The userinformation 113 includes coordinates (specified coordinates) specifiedby the user input and time (specified time) at which the input was made.For example, in the case of using a touch panel, the user information113 may be input by means of the user making an action such as tapping(using a finger), at a given time, a point on the display screencorresponding to the subject region to be specified, drawing a circleenclosing the subject region to be specified, and flicking the subjectregion to be specified.

The input receiving unit 105 may be an interface for reading thespecified time or the specified coordinates included in the userinformation 113 directly or through a communication path, in response tothe user operation using a mouse or a touch on a touch panel on whichthe video 110 is displayed. Here, the number of user inputs does notneed to be only once. For example, a plurality of object can be selectedby touching different regions. In addition, for userfriendliness in thecase of selecting all of the objects in a given display screen or avideo, an icon “Select all objects in display screen” or an icon “Selectall objects in video” may be separately provided on a video displayscreen.

As an example of the input receiving unit 105, FIG. 13 shows an examplein which the user information 113 is input through a touch panel. Inthis case, the coordinates (specified coordinates) specified by the userand the time (specified time) in the video 110 are obtained as userinformation 113, and are received by the input receiving unit 105. In asubsequent step, the specified coordinates and the specified time areverified with region information. In this case, information that “Thesubject region 402 was selected” is obtained.

With reference to FIG. 8 again, in Step S306, the display prioritycalculating unit 106 calculates display priority for each of thecomments and subject regions in the video 110, based on the subjectregion information 111, the comment information 112, and the userinformation 113. The display priority calculating unit 106 determines,for each of the comments and the subject regions, the display prioritiesfor generating a comment-provided video in which a user-specified region(that is calculated based on the user information 113 received by theinput receiving unit 105) is not hidden by any other comment regions.Exemplary rules which can be used as rules according to which thedisplay priority calculating unit 106 calculates the display prioritiesof the respective regions include “calculating, for a user-specifiedregion, a display priority higher than display priorities of the otherregions, based on the subject region information 111, the commentinformation 112, and the user information 113 obtained by the dataobtaining unit 104, and calculates a higher display priority for a newercomment among comments”.

Each display priority is used by the comment superimposing methoddetermining unit as a standard for determining a method forsuperimposing the comment (comment region), FIG. 14 is an illustrationshowing a concept of comment superimposition and display based on thedisplay priorities. FIG. 14 shows the subject regions 400 to 403 and thecomment regions 500 to 502. Here, for explanation, the relationshipbetween a comment and a subject region to which the comment is providedis referred to as “association”. The comment region 500 is associatedwith the subject region 401, the comment region 501 is associated withthe subject region 402, and the comment region 502 is associated withthe subject region 403. The association information between the commentregion and the subject region is stored in the comment information 112.The subject regions 400 to 403 and the comment regions 500 to 502 arepartly overlapped with each other, and only pixels superimposed on topof the other superimposed regions are displayed.

In the example shown in FIG. 14, the background region (the subjectregion 400) having the lowest display priority is displayed at therearmost position, and the subject regions 400 to 403 and the commentregions 500 to 502 are superimposed and displayed in a layered structurein a given order. This display order is determined as display prioritiesP. A region having a lower display priority is moved backward by anotherregion superimposed thereon and then disappears. More specifically, FIG.14 is a diagram showing a superimposition and display example in whichthe display priorities P of the respective regions are shown accordingto P₄₀₁>P₄₀₂>P₅₀₁>P₅₀₀>P₄₀₃>P₅₀₂>P₄₀₀. Accordingly, when the order ofmagnitudes of the display priorities P of the respective regions ischanged, the superimposition order of the regions is also changed. Forexample, when the subject region 401 and the subject region 402 are theregions that a user wants to see without occlusion, it is possible torealize comment display in which these regions are not occluded byproviding display priorities P higher than those of the other regions asthe simplest manner.

Hereinafter, the term “superimposition” in this disclosure is referredto as superimposition of regions on an image. The display prioritycalculating unit 106 verifies the coordinates and time in the userinformation 113 obtained by the input receiving unit 105 and the subjectregion information 111 obtained by the data obtaining unit 104, todetermine the selected one or more of the regions.

As shown in FIG. 10B, the “region information” may further include a“region attribute”. The region attribute is information indicatingwhether each subject region is a still region or a moving region. Inthis embodiment, a still region is assigned with 0, and a moving regionis assigned with 1. For example, in a video in which people are doing asport or a video in which people are dancing, most of subjects which maybe selected by a user are moving objects. Thus, assignment of suchattributes makes it easy to calculate display priorities. In otherwords, the display priority calculating unit 106 may determine a region(a still region) having a region attribute of 0 to be a backgroundregion, and always assign a smallest display priorities P thereto.

Furthermore, the “region information” may include information indicatingan occlusion relationship between subject regions in a video. Forexample, information of subject regions in an occlusion relationship isstored with video time or a frame number. More specifically, in the caseof a video displayed in an example in FIG. 12, it is good to storeinformation indicating that a subject C occludes a subject A at videotime “00:00:19”, for all video time (the alphabets A to C assigned tothe subjects are the same as those in FIG. 9).

Alternatively, information indicating a similar occlusion relationshipmay be calculated based on region information obtained by the dataobtaining unit 104. In this case, it is possible to determine theocclusion relationship by, for example, calculating a motion vector neara boundary between regions in advance, and determining that a firstregion in which a larger number of motion vectors or moving trajectoriesdisappears or stops in a focused certain time zone in video timecompared to a second region is occluded by the second region in the timezone. When no such information indicating an occlusion relationship isprepared, occlusion information is calculated.

The use of occlusion information obtained as described above makes itpossible to generate a video in which one or more regions important to auser are easy to see.

In the case of FIG. 15A, comments provided to given subjects move closerto each other, overlap making an occlusion relationship, and then passthrough each other. This display is not easy to see for the user. Inview of this, the comment superimposing method determining unit 107determines a superimposing method for displaying a video by temporarilystopping comment regions of comments corresponding to subject regionswhich make an occlusion relationship at positions before a start of anocclusion period in which they are in the occlusion relationship,releasing the temporary stop after the occlusion period to cause thecomments to re-start tracking the subject regions. In this way, acomment-provided video as shown in FIG. 15B is displayed on a display120. In the video, the comment regions of the comments corresponding tothe subject regions which make the occlusion relationship temporarilystop at the positions before the start of the occlusion period in whichthe subject regions are in the occlusion relationship, and after theocclusion period, the comments re-start tracking the subject regions. Inthe display examples of FIGS. 15A and 15B, the occlusion starts at t=1,ends at t=2, the positions of the comment provided to the subjectregions at t=1 are maintained also at t=2.

The way of display can prevent one of the comments provided to thesubject regions from being occluded by another one of the commentsprovided to the other subject regions, especially for the user who wantsto see the comments provided to both the subject regions which make theocclusion relationship. In this way, it is possible to display thecomments in a simple easy-to-see manner.

As another display method, the comment superimposing method determiningunit 107 determines a superimposing method for displaying thecomment-provided video in which balloons including comments respectivelycorresponding to the subject regions which make the occlusionrelationship are integrated into one balloon during the occlusionperiod, and after the occlusion period, the integrated balloon isdivided into separate balloons respectively including the comments tocause the comment regions to re-start tracking the subject regions.FIGS. 16A and 16B show examples realized in this way. In the video onthe display 120 in each of the examples, the balloons including thecomments corresponding to the subject regions which make the occlusionrelationship are integrated into the one balloon during the occlusionperiod, and after the occlusion period, the one balloon is divided intoseparate balloons respectively including the comments to cause thecomment regions to re-start tracking the subject regions. It is to benoted that the comment superimposing method determining unit 107 maydisplay these comments in a horizontal layout or in a vertical layout inone comment region integrated as the balloon. In FIGS. 16A and 16Bsimilarly to the examples of FIGS. 15A and 15B, the occlusion starts att=1, and ends at t=2. In each of examples in FIGS. 16A and 16B, simply,a balloon having a size corresponding to the total size of the twoballoons is displayed at the center position of these two balloons atthe start of the occlusion. As shown in FIG. 16A, it is also good todetermine the superimposing method for displaying the balloon having atip which tracks the subject region. Alternatively, in order to preventthe end of the balloon to have a complex shape when these two subjectregions pass through each other, it is also good to determine asuperimposing method for displaying the balloon by temporarily stoppingthe balloon integrated at t=1 and keeping the balloon until t=2 as shownin FIG. 16B similarly to the example of FIG. 15B.

In this way, it is possible to allow a user who wants to see both thecomments provided to the subject regions which make the occlusionrelationship, to see the comments in a simple, easy-to-see displaymanner. Furthermore, it is possible to show that the subjects with thecomments are temporarily in the occlusion relationship, in a moreemphasized display manner.

As another display method, the comment superimposing method determiningunit 107 determines a superimposing method for displaying thecomment-provided video in which only one of comments which correspondsto an occluding subject region closer to a camera used to capture thevideo among the subject regions which make the occlusion relationship isdisplayed during the occlusion period, and after the occlusion period,all of the comments are displayed. In this way, as shown in FIG. 17B,only the comment corresponding to the occluding subject region closer tothe camera used to capture the video among the subject regions whichmake the occlusion relationship may be displayed during the occlusionperiod, and after the occlusion period, all of the comments may bedisplayed.

In this way, it is possible to provide intuitive display, as shown inFIG. 17B, which is easier to see, and virtually realizing in the video arelationship between the subjects in a real scene, compared to theballoons at the time the subjects pass through each other in FIG. 17A.

Furthermore, as shown in FIG. 18A, the comment superimposing methoddetermining unit 107 may perform a display method for displaying one ofcomments which corresponds to an occluding subject region in asemi-transparent display manner on the display 120 in an occlusionperiod, and, after the occlusion period, re-start displaying all of thecomment in a non-semi-transparent display form.

In this way, it is possible to realize simple display while maintainingthe visibility of the subjects as in the earlier-described method.

Alternatively, the comment superimposing method determining unit 107 maysuperimpose and display the comment corresponding to the occludedsubject region on the occluding subject in a transparent display form,and, after the occlusion period, re-start displaying all of the commentsin a non-semi-transparent display form. In this way, as shown in FIG.18B, the comment corresponding to the occluded subject region may besuperimposed and displayed on the occluding subject region in thetransparent display form, and, after the occlusion period, display ofall the comments may be re-started in the non-semi-transparent displayform.

In this way, it is possible to provide display which emphasizes therelationship between the subjects in the real scene, and more intuitiveto the user, while maintaining the visibility of the comments.

In addition, it is possible to show that the subjects with the commentsare temporarily in the occlusion relationship, in an emphasized displaymanner, irrespective of which one of the display methods in FIGS. 18Aand 18B is taken.

The different balloons in each frame in each of FIG. 15A to FIG. 18Bhave different textures for the purpose of increasing visibility andsimplifying the explanation here, but the textures do not always thesame as actual ones. In reality, for example, a higher visibility ofcharacters is obtained when all of the balloons on which the charactersare displayed do not have any texture.

There are some factors that affect the look of a comment-provided videoor the visibility of each region. If considering communication bycomments as important, a user can enjoy communication on the video morewhen comment regions are basically superimposed on subject regions. Inreality, many comment-provided video services employ display methods inwhich comment regions occlude subject regions.

Furthermore, it is considered that a user can easily see comments when acomment region including a comment that is newly submitted and having anewer submission date and time is superimposed at a frontward positionand when a comment region including a comment that is newer in videotime is superimposed on an older comment region including an oldercomment which is hidden behind the newer comment.

In addition, a comment region associated with a more frontward subjectregion, in other words, a comment region displayed at a frontwardposition is considered to be intuitively easy to see.

It is the user's intention that is more important than theabove-described factors. A comment-provided video that is most easy forthe user to see is generated by displaying a subject region or a commentregion selected according to user's intention at a foremost positionfree from occlusion.

The above-described conditions are organized in a table in FIG. 19. Forexample, when there is no user input, in other words, when neither asubject region nor a comment region is selected (in (A) in the table inFIG. 19, the display priority calculating unit 106 determines displaypriorities for superimposing the subject region including a backgroundregion below the comment region ((a) in (A)). In addition, the displaypriority calculating unit 106 determines display priorities forsuperimposing a comment region having an earlier video time at abackward position ((b) in (A)). The display priority calculating unit106 may determine display priorities for displaying a comment regionhaving a newer real submission time in addition to newer video time at afrontward position.

When a subject region is selected according to a user input and thecomment region is not selected (in (B) in FIG. 19), the display prioritycalculating unit 106 determines display priorities for displaying theselected subject region at a foremost position ((a) in (B)), inaddition, the display priority calculating unit 106 determines displaypriorities for the unselected subject region and comment regionaccording to the standards shown in (A) ((b) in (B)).

When a comment region is selected according to a user input and thesubject region is not selected (in (C) in FIG. 19), the display prioritycalculating unit 106 determines display priorities for displaying theselected comment region at a foremost position ((a) in (c)). Inaddition, the display priority calculating unit 106 determines displaypriorities for the unselected subject region and comment regionaccording to the standards shown in (A) ((b) in (C)). However, thedisplay priority calculating unit 106 may determine display prioritiesfor displaying the subject region corresponding to the selected commentregion at a position that is frontward next to the selected commentregion ((a′) in (C)).

When both of the comment region and the subject region are selected asuser inputs ((D) in FIG. 19), the display priority calculating unit 106may determine display priorities for displaying the selected commentregion at a foremost position ((a) in (D)). In addition, the displaypriority calculating unit 106 determines display priorities fordisplaying the selected subject region that is frontward next to theselected comment region at a foremost position ((b) in (D)). Inaddition, the display priority calculating unit 106 determines displaypriorities for the unselected subject region and comment regionaccording to the standards shown in (A) ((b) in (D)). However, thedisplay priority calculating unit 106 may determine display prioritiesfor displaying the subject region corresponding to the selected commentregion at a position that is frontward next to the selected subjectregion ((a′) in (D)). In addition, the display priority calculating unit106 may determine display priorities for displaying the comment regioncorresponding to the selected subject region at a position that isfrontward next to either the selected subject region or the subjectregion corresponding to the selected comment region ((b′) in (D)).

The display priority calculating method according to the rules based onthe above-described conditions makes it possible to generate acomment-provided video reflecting users' intention. As an exemplarycalculating method, it is possible to perform processes based on adisplay priority calculation flow as shown in FIG. 20 (each “displaypriority” is briefly written as a “priority” in the drawing). It is tobe noted that the association between a comment region and a subjectregion may be stored in advance in a comment information storage server,or a subject region at coordinates corresponding to base coordinates ofa comment region including a comment at a display time may be searchedout from the subject region information 111, using display time and basecoordinate information of the comment included in the commentinformation 112.

Whether there is a user input through the input receiving unit 105 ischecked for each frame or a particular frame first. When there is nouser input, the display priority calculating unit 106 calculates displaypriorities for realizing comment display in a conventional approach,according to the flow shown in processes included in (A) in FIG. 20.

When there is a user input, as described earlier, sets of processes areswitched depending on whether one or both of the subject region and thecomment region are selected. This is performed to determine the regionthat is to be preferentially displayed when the subject region and thecomment region or comment regions overlap with each other at somepixels. By performing the process sets (B), (C), and (D) in FIG. 20,display priorities are calculated based on the above-described rules.

The order of processes (1) to (3) in the process set (A) may beexchanged according to a user's taste or the like.

FIG. 21A to FIG. 21F show examples of superimposition and display ofcomment regions and subject regions obtainable as results of calculatingdisplay priorities according to rules shown in FIG. 19 or processingflows in FIG. 20 for cases in which different regions are selected.

FIG. 21A shows a video 110 in which subject regions 400 to 403 includinga background are shown. The subject regions 400 to 403 are shown by thesame subject region information 111 as shown in FIG. 10A. Furthermore,as in the case of FIG. 11A, FIG. 21B shows a display example of a videoin which the comment regions 500 to 502 are superimposed at a giventime. Here, the comment information 112 used to display the commentregions 500 to 502 is the same as the comment information 112 in FIG.11B. FIG. 21B shows a general comment display example in which thecomment regions 500 to 502 are displayed frontward than the subjectregions 400 to 403 including the background. For this reason, thesubject regions 401 to 403 other than the background are occluded. Acomment region having a newer display time is displayed at a morefrontward position (the comment region 500 is the newest commentregion). In this embodiment, unless any instruction is provided from auser, a higher display priority is assigned to a subject region to bedisplayed at a frontward position (closer to the viewer in the depthdirection), and a higher display priority (for decreasing thepossibility of occlusion by another comment region) is assigned to acomment region having a newer display time. In addition, unless anyinstruction is provided from a user, each comment region is assignedwith a display priority higher than a display priority of a subjectregion, and, among all of the regions, a background region has a lowestdisplay priority.

Here, as described earlier, the comment region 500, the comment region501, and the comment region 502 correspond to the subject region 401,the subject region 402, and the subject region 403, respectively. Thus,the priorities calculated in the case of FIG. 21B are representedaccording to P₅₀₀>P₅₀₁>P₅₀₂>P₄₀₁>P₄₀₂>P₄₀₃>P₄₀₀. FIG. 21B is a displayexample corresponding to the process set (A) in FIG. 19 or the processset (A) in FIG. 20.

In addition, FIG. 21C shows a display example of a video that is outputby the comment-provided video generating apparatus 100 when the subjectregion 402 is selected by the user. FIG. 21C is a display examplecorresponding to the process set (B) in FIG. 19 or the process set (B)in FIG. 20. FIG. 21C shows that the subject region 402 is displayed at aforemost position, a comment region 501 corresponding to the subjectregion 402 is superimposed next, and the other regions are superimposedbehind the comment region 501. The display priorities calculated in thiscase are represented according to P₄₀₂>P₅₀₁>P₅₀₀>P₅₀₂>P₄₀₁>P₄₀₃>P₄₀₀.

In addition, FIG. 21D shows a display example of a video that is outputby the comment-provided video generating apparatus 100 when the subjectregion 502 is selected by the user. FIG. 21D is a display examplecorresponding to the process set (C) in FIG. 19 or the process set (C)in FIG. 20. FIG. 21D shows that the subject region 502 is displayed at aforemost position, a subject region 403 corresponding to the subjectregion 502 is superimposed at a next frontward position, and the otherregions are superimposed behind the subject region 403. The displaypriorities calculated in this case are represented according toP₅₀₂>P₄₀₃>P₅₀₀>P₅₀₁>P₄₀₁>P₄₀₂>P₄₀₀.

In addition, FIG. 21E shows a display example of a video that is outputby the comment-provided video generating apparatus 100 when the subjectregion 402 and the comment region 502 are selected by the user. FIG. 21Eis a display example corresponding to the process set (D) in FIG. 19 orthe process set (D) in FIG. 20. FIG. 21E shows that the comment region501 corresponding to the subject region 402, the subject region 403corresponding to the comment region 502, the subject region 402, and thecomment region 502 are sequentially superimposed in the listed order (inother words, the comment region 502 is superimposed at the foremostposition). FIG. 21E also shows that the other regions are superimposedbehind the comment region 501. The display priorities calculated in thiscase are represented according to P₅₀₂>P₄₀₂>P₄₀₃>P₅₀₁>P₅₀₀>P₄₀₁>P₄₀₀.

Here, FIG. 21F shows a display example of a video that is output by thecomment-provided video generating apparatus 100 when all of the subjectregions 401 to 403 are selected by the user. FIG. 21F is a displayexample corresponding to the process set (B) in FIG. 19 or the processset (B) in FIG. 20. FIG. 21F shows that all of the comment regions aresuperimposed at a position backward than the subject regions 401 to 403and frontward than the subject region 400 which is a background. Thedisplay priorities calculated in this case are represented according toP₄₀₁>P₄₀₂>P₄₀₃>P₅₀₀>P₅₀₁>P₅₀₂>P₄₀₀.

According to the above-described rules, the display priority calculatingunit 106 calculates the display priorities for the comment regions 500to 502 in the video 110 and the subject regions 400 to 403 including thebackground, based on the subject region information 111, the commentinformation 112, and the user information 113. The values indicatingdisplay priorities may be any ones which can represent priorityrelationships between the regions. For example, in the case of sevenregions as described above, it is good to assign the background whichshould have a lowest display priority 0 as a value indicating thedisplay priority thereof, and sequentially assign the other regions 1,2, 3, . . . as values indicating a display priority thereof in thedescending order of the display priority.

As described above, it is a user's selection that affects most thepriority order indicated by the display priorities. The depthrelationships between subject, comment submission time and/or displaytime, etc. have also been listed as factors that affect the displaypriorities. In addition to these factors, for example, when the commentinformation 112 includes users' ID who made comments, this may affectthe display priority of a comment with the user ID specified in advanceby the user.

In the earlier description, when a plurality of regions have the samedisplay priority, the priority order is determined at random as shown inthe process set (A) [2] (1)-(4). In stead, it is also good to displaythe fact that a plurality of regions have the same display priority onthe display, and allow the user to select one or more of the regionswhich are to be displayed preferentially. In addition, when there aretwo comments which are to be overlapped have different numbers ofcharacters, the comment region including the comment having a fewernumber of characters may be made smaller and displayed in front of theother in order to increase the visibility of both the comments.

With reference to FIG. 8 again, in Step S307, the comment superimposingmethod determining unit 107 determines a region superimposing method forgenerating a comment-provided video 110 in which the subject regions andthe comment regions are composed, based on the display prioritiescalculated by the display priority calculating unit 106.

More specifically, the region superimposing method calculated here is amethod for calculating a region superimposing order in which the regionsare sequentially superimposed in a descending order of the displaypriorities calculated by the display priority calculating unit 106. Inrelation to the descriptions with reference to FIG. 21A to FIG. 21F,control for preventing the user-specified region from being occluded byanother one of the comment regions or subject regions is performedaccording to the region superimposing order based on the displaypriorities.

Alternatively, the comment superimposing method determining unit 107 maycalculate display transparencies for comment regions based on thedisplay priorities calculated by the display priority calculating unit106, instead of calculating the region superimposing order. Morespecifically, in the case of calculating transparencies for pixels incomment regions which are to be overlapped, the comment superimposingmethod determining unit 107 sets a transparency=0 (zero transparency) tothe pixel having a highest display priority among those calculated bythe display priority calculating unit 106, and sets a transparency>0 toeach of the pixels having a lower display priority. These transparenciescan be determined based on ratios of display priorities etc. In order toincrease visibility, the entire comment region may be displayedtransparent, instead of the overlap pixel, in this case, to increase thevisibility in the transparent display, the colors of the region and thecharacters are automatically converted into white and block,respectively, at the time of display irrespective of whether or not thecolors of the comment region and comment characters are specified inadvance.

In Step S302, the image composing unit 102 generates a composedcomment-proved video by composing comments to be displayed in the video110 according to the region superimposing method determined by thecomment superimposing method determining unit 107. When the commentsuperimposing method determining unit 107 calculates a commentsuperimposing order, the image composing unit 102 generates acomment-provided video by composing in layers comments to be displayedin the video 110. Alternatively, when the comment superimposing methoddetermining unit 107 calculates display transparencies for the commentsinstead of the comment superimposing order, the image composing unit 102may determine pixel values corresponding to the regions according to thetransparencies and generate a comment-provided video by composing thecomments in the video 110.

In Step S303, the image output unit 103 outputs, onto the display 120 orthe like, the comment-provided video composed by the image composingunit 102.

The comment-provided video generating apparatus 100 in this embodimentincludes the input receiving unit 105 which receives the userinformation 113. However, for example, in the case where a TV or displayconnected through a network is used and when no user inputs(specification of time and coordinates) are expected or absence of userinputs is known in advance, the comment-provided video generatingapparatus does not need to include the input receiving unit 105. Morespecifically, as exemplary display forms, it is also good to applyautomatically other rules for increasing visibility of subjects insteadof using user input information.

One of these rules is (a) to assign only the subject region in theforemost row a display priority higher than those of comment regions.

The other rule is (b) to assign all subject regions display prioritieshigher than those of all comment regions.

The comment-provided video generating apparatus in this embodiment mayinclude a user information obtaining unit in stead of or in addition tothe input receiving unit 105. FIG. 22 is a block diagram showing astructure of a comment-provided video generating apparatus 100Aincluding a user information obtaining unit. The comment-provided videogenerating apparatus 100A basically has the same structure as that ofthe comment-provided video generating apparatus 100 shown in FIG. 6, andincludes the user information obtaining unit 114 instead of the commentsuperimposition information generating unit 101. The commentsuperimposition information generating unit 101A includes the userinformation obtaining unit 114 that is not included in the commentsuperimposition information generating unit 101. The user informationobtaining unit 114 obtains specified coordinates and specified timeinstead of pre-stored user information 113, from an informationdistribution source (a user ID information server) or the like through awired or wireless communication path.

In addition, the user ID information server may store tastes of each ofusers who are browsing web pages or information such as a behaviorhistory on the Internet of the user. The user information obtaining unit114 may obtain information such as the user's tastes and behaviorhistory instead of obtaining user information 113 through an interface,and estimate the user information 113 based on the obtained information.With this structure, it is possible to automatically display acomment-provided image that matches the user's tastes without receivingany intuitive inputs from the user.

More specifically, for example, the display priority calculating unit106 determines whether or not a comment provided in a video that iscurrently being viewed by the user includes a character string (such asthe name of a particular subject) registered in advance by the user andstored in the user ID information server or a character stringcorresponding to an article which is in a Social Networking Service(SNS: Facebook or the like) or on the Internet and is shared by users orselected by means of a user pressing a Like button. If the determinationresult is true, the display priority calculating unit 106 may determinethat the comment region of the comment and the subject regioncorresponding to the comment are selected by the user, increase thedisplay priorities of the respective regions, and display these regionsat more frontward positions.

In particular, as shown in FIG. 23, a user who is viewing a video mayperform another activity on the Internet simultaneously with the viewingor before or after the viewing. For example, a user may read an articleon the Web or watch an SNS while viewing a video. At this time, when theuser shares an article in an SNS or the Internet and/or presses a Likebutton for the article as shown in (a) of FIG. 23, the article to whichthe Like button pressed or the character string corresponding to theshared article may be added to the user ID information server while thevideo is being viewed as shown in (b) of FIG. 23. FIG. 23 shows a casewhere a character sting “yy” replaces a character string “xx” and isadded to the information in the user ID information server. When theuser ID information is updated based on a user's behavior history on theInternet, it is possible to automatically provide display reflecting theuser's tastes by taking a display method for sequentially modifying thesuperimposition and display order as shown in (b) of FIG. 23 based onthe new user ID information in the user ID information server, withoutrequiring the user to perform any operation such as a touch in order tosequentially modify the user information in connection with the video.In other words, the comment region of the comment corresponding to thesubject region corresponding to the character string “xx” is displayedat a frontward position.

The user ID information server also stores information about othercomment providers, and thus user ID and terminal numbers of the commentproviders may also be used in the display priority calculating process.For example, by storing inter-user friendship in the SNS or followstates in association with the user ID in the user ID informationserver, the display priority calculating unit 106 can perform weightingfor increasing the values of the display priorities for commentsprovided by a provider who is one of friends of the user who iscurrently viewing a video or comments provided by a provider who isfollowed by the user who is currently viewing the video. In this case,as in the earlier-described example, by performing the display methodfor sequentially modifying the superimposing and display order accordingto the friendship in the current SNS or a change in the follow states,it is possible to automatically perform display reflecting the user'stastes.

In addition, the user ID may be directly specified instead of the userspecification of the subject region or the comment region. In this case,the display priority calculating unit 106 may perform the process set(C) in FIG. 19 or the process set (C) in FIG. 20, assuming that all ofthe comments provided by a specific user having the specified user IDare selected by user inputs.

Furthermore, as shown in FIG. 24, the user ID information server maystore a viewing history of the user, together with the user ID. The userdoes not always watch the same video only once. At this time, when nocomment on the video is added by any other user between plural viewingtimes of the video by the user, the video with the same comments isdisplayed in such a manner that an already-read comment is superimposedat the foremost position at both of the plural viewing times.

In view of this, the user information obtaining unit 114 is configuredto obtain the viewing history of the user who is viewing acomment-provided video, and determine whether or not the user hasalready watched the same comment-provided video before. Furthermore, thedisplay priority calculating unit 106 decreases the display priority ofthe region displayed at the foremost position at the previous viewingtime, and superimposes and displays the region at a backward position,thereby preventing the same comment-provided video is displayed as seenbefore as described earlier.

In FIG. 24, (a) shows first viewing of a video by the user. At thistime, viewing time zone and information about the video are written intothe user ID information server. Information about the display prioritiesin the video at the first viewing may be additionally written into theuser ID information server.

As shown in (b) in FIG. 24, when the user views the video next, thedisplay priority calculating unit 106 makes an inquiry to the user IDinformation server to determine whether or not the current video isalready written in the user ID information server, thereby determiningwhether or not the video is an already viewed one. When the currentvideo is an already viewed one, with reference to the informationindicating the display priorities at the last-time viewing, the displaypriority calculating unit 106 assigns the region displayed at theforemost position last time the lowest display priority. Here, thecomment-provided video generating apparatus 100 adds the viewing historyto the user ID information server as shown in (C) of FIG. 24 similarlyto the first-time viewing of the video.

In this way, in the plural viewing times, the superimposing order ofregions to be displayed is updated for each of the viewing times. Thus,even if the same user views the video having substantially the samecontent plural times, the user can enjoy the video modified tosuperimpose the comments differently every viewing time, instead of thesame video in which the same comment is fixedly displayed at theforemost position.

Here, the comment information 112 does not always need to store the basecoordinates of all the frames. For example, when region informationincludes motion information between pixels in regions in adjacent framesor motion information between the entire regions, it is possible todisplay a comment tracking a region or pixels between the regions orpixels, based on the base coordinates at a comment starting time, andthe pre-calculated pixel motion or region motion (FIG. 25A, and FIG.25B).

Even in a particular case where subject regions have variable sizes,natural tracking can be realized by calculating, for comment regions,base coordinates which maintain depth relationships with the centers ofsubject regions included in region information. In other words, the basecoordinates of the comment regions are calculated based on the size ofthe subject region corresponding to the comment in a comment startingframe and the pixel positions of the base coordinates at the commentstarting time on the subject region, so as not to relatively change thepositions of the comment regions with respect to the subject regions.For example, it is also good to calculate base coordinates of commentregions at and after a current frame by, for example, normalizing thedistances from the base coordinate positions of the comment regions tothe centers of the subject regions to which comments are provided, usingthe sizes of the subject regions.

In addition, fast forward, preview, rewind, skip etc. may be performedon a comment-provided video, in addition to reproduction at a normalvideo rate. Particularly when a large number of comments are provided toa video, a user who normally understands the comments at a normal ratehas difficulty in recognizing the comments when the video is reproducedat a simply increased frame rate when performing normal fast forward orrewind or a jump between frames.

In view of this, display forms are modified in the case where viewingmethods such as fast forward, preview, rewind, a skip etc. areperformed. FIG. 26 shows such modification examples. In display examples(b), (d), and (f) (modified from display examples (a), (c), and (e),respectively), only the foremost comment region is superimposed anddisplayed in each of frames. More specifically, it is assumed here thatone of the display forms as in (a), (c), and (e) is provided at the timeof normal reproduction. When one of viewing methods such as fastforward, preview, rewind, a skip, etc, is performed while one of thedisplay forms (a), (c), and (e) is provided, the display form ismodified to a corresponding one of the display forms as shown in (b),(d), and (f).

The user needs to understand information quickly when one of the viewingmethods such as fast forward and rewind is performed. By displaying onlythe important comment as in each of the above cases, it is possible toprovide display which reflects the user's tastes, includes a reducedamount of comment information to be understood by the user, and is easyto see even when such fast forward and rewind is performed.

As described above, according to Embodiment 1, it is possible togenerate a comment-provided video in which comment regions aresuperimposed and displayed, based on rules for increasing displaypriorities for one or more comment regions and/or subject regionscorresponding to time and coordinates specified in the video which areincluded in user information. In this way, it is possible to display thecomment-provided video in such a userfriendly manner that comment andsubject regions which are important to the user are not occluded bybeing superimposed behind another comment region although the othercomment and subject regions are superimposed behind another commentregion.

Embodiment 2

The comment-provided video generating apparatus 100 in Embodiment 1receives the subject region information 111 obtained in advance asinputs to the data obtaining unit 104 and processes the inputs. However,instead of the subject region information 111, other subject regioninformation 111 may be generated in a comment-provided video generatingapparatus. In view of this, the comment-provided video generatingapparatus according to this embodiment includes a subject regioninformation generating unit which generates subject region information.In this case, the video distributing system according to Embodiment 2is, for example, as shown in a configuration example in FIG. 27. Inother words, the video distributing system has an advantage of beingmore versatile because it does not require a region information serverfor storing subject region information that is included in theconfiguration in Embodiment 1 as shown in the example of FIG. 1, FIG. 3,or FIG. 4.

FIG. 28 shows a structure of the comment-provided video generatingapparatus 900 in Embodiment 2. The comment-provided video generatingapparatus 900 includes a comment superimposition information generatingunit 901, an image composing unit 102, and an image output unit 103. Thecomment-provided video generating apparatus 100 includes the commentsuperimposition information generating unit 901 which performs a processdifferently from the comment superimposition information generating unit101 included in the comment-provided video generating apparatus 100 inEmbodiment 1 shown in FIG. 6.

More specifically, the comment superimposition generating unit 901includes a data obtaining unit 902 and a subject region informationgenerating unit 903 instead of the data obtaining unit 104 inEmbodiment 1. The data obtaining unit 902 obtains a video 110 andcomment information 112. The subject region information generating unit903 receives the video 110 as an input, and generates subject regioninformation. The structure in this embodiment makes it possible tocalculate the subject region information (a region attribute, a positionattribute, division information, etc.) from the video 110, therebygenerate the comment-provided video without any input of subject regioninformation.

The data obtaining unit 902 differs from the data obtaining unit 104 inEmbodiment 1 only in the point of whether or not to receive the subjectregion information 111, and thus no detailed descriptions are providedhere.

The subject region information generating unit 903 performs a uniquesubject region information generating process which may involve, forexample, a motion-based division method. More specifically, the subjectregion information generating unit 903 may apply an approach fordividing a video into regions based on pixel motions as explained inPatent Literature 6. This provides an effect of being able to correctlydivide subject regions in the video into subject region sets eachincluding a similar motion. At that time, the division is performedbased on a distance between motion trajectories each showing a joint ofeach subject in the video. Thus, it is possible to extract subjectregions including the same subject precisely even if the subject is amoving body, such as a person, whose posture changes significantly inthe video. In addition, for example, between two adjacent subjectregions, it is possible to determine that a first subject region isoccluded by a second subject region in a time zone in video time whenthe number of motion vectors which disappears or stops around a boundarybetween the adjacent subject regions in the time zone is larger in thefirst subject region than in the second subject region. In this way,focusing on the disappearance of the motion trajectories, it is possibleto determine information indicating the relative positions of theadjacent subject regions.

Here, the comment information 112 in this embodiment does not alwaysneed to store the base coordinates of all the frames. For example, whenthe base coordinates at a comment starting time, it is possible todisplay the comment with the subject region to which the comment isprovided and cause the comment to track the subject region, based on amotion of the same subject region between frames calculated by thesubject region information generating unit 903 (FIG. 25A, FIG. 25B).

Alternatively for example, in order to maintain the depth relationshipbetween the center of the subject region and the comment basecoordinates, it is good to provide an object-tracking comment coordinategenerating unit which calculates base coordinates of a comment region inthe frame and the following frames, from the subject region informationand the comment information 112 as inputs.

It is to be noted that each of the structural elements of the subjectregion information generating unit 903 may be configured as softwaresuch as a program which is executed on a computer, or may be hardwaresuch as an electric circuit or an integrated circuit.

The subject region information generating unit 903 may perform divisioninto regions using measurement data, or the like. For example, it ispossible to perform division into regions more precisely by combiningthe above-described division approach (Patent Literature 6) usingpair-wise distances with an additional process of decreasing a pair-wisedistance between regions having similar measurement data items andincreasing a pair-wise distance between regions having dissimilarmeasurement data items. Furthermore, the use of such measurement datamakes it possible to calculate depth relationship between subjectregions to some extent. Therefore, the display priority calculating unit106 may use the approach for calculating the local display priorities,or may use the approach involving ranking of display priorities of allthe regions.

The video distributing system in this embodiment includes a serverapparatus which is held and managed by an information distributionsource, a plurality of personal computers (PCs) which can performnetwork communication, a plurality of mobile terminals (mobile phones,smart phones, tablet PCs, etc.) which can also perform networkcommunication, and a wired or wireless communication line.

The server apparatus, the PCs, and the mobile terminals are connecteddirectly or through a relay base station to the communication line wiredor wirelessly, and can mutually communicate information through thecommunication line.

In addition to the effect obtainable in Embodiment 2. Embodiment 2 makesit possible to detect subject regions through division into regions evenif the subject regions included in the video are unknown as describedabove.

The comment-provided video generating apparatus in each of Embodiments 1and 2 can be effectively used as an input supporting apparatus. Anexample is shown below.

The video distributing system described earlier in this disclosure cannot only allow a user to view a comment-provided video but also allowthe user to additionally input his or her comment. The input device maybe a tablet, a PC, or the like, and the user may input and provide acomment sentence to the video using a method represented by alreadyproposed approaches such as the one shown in Patent Literature 1.

FIG. 29A is a block diagram showing a configuration of a systemincluding an input device and an input supporting apparatus. FIG. 29B isa flowchart of processes executed by the input supporting apparatus.Here, the input supporting apparatus described here corresponds to adisplay control apparatus for performing display control at the timewhen the comment is input, and as shown in FIG. 29A, receives inputstart information indicating that a comment is to be input by a user,from the input device. See (a) of FIG. 30A showing an example where acomment icon 301 which can be selected when a user provides a comment isdisplayed on a display screen of an input device. When the user pressesthe comment icon 301, the input device may transmit the input startinformation to the input supporting apparatus. In FIG. 30A, (b) showsanother example where the input device transmits input start informationto the input supporting apparatus by receiving a particular user actionsuch as an upward swipe from a lower end of the display screen.

When the input supporting apparatus receives the input start informationfrom the input device, the reproduction of the video 110 may beautomatically and temporarily stopped, and then the comment may besubmitted during the temporary stop. Alternatively, the reproduction ofthe video 110 may be temporarily stopped by a user operation on thedisplay screen.

The input supporting apparatus receives the input start information asdescribed above, and thus can know that the user intends to input thecomment before the user actually inputs the comment.

Here, when the comment is added to the comment-provided video in which alarge number of comments is already provided, there is a possibilitythat a subject region or a comment region for or in which the userintends to add his or her comment is occluded by another comment regionor subject region. Such occlusion should preferably be prevented whenthe user adds his or her comment smoothly for or in a target region. Toprevent such occlusion, the input supporting apparatus modifies adisplay order in advance when receiving the input start information.

A specific example is shown in (a) of FIG. 30B. For example, when theuser inputs a comment for a subject 302 located at the left side, thereis a possibility that the user has difficulty in inputting his or hercomment because the subject 302 is occluded by existing comments as in(a) of FIG. 30B which are displayed in the same manner as in viewingtime. In this case, by displaying the subject at a foremost position asshown in (b) of FIG. 30B, it is possible to allow the user to easilyinput his or her comment for the particular subject.

This can be performed by estimating the subject to which the userintends to input his or her comment. In most cases, users generallyinput comments for particular subjects. For this reason, the simplestmethod is to display all subject regions at positions in front of allcomment regions.

Furthermore, it is also good to register in advance informationreflecting users' tastes in a user ID information server, and to displayonly subject regions of subjects included in the registered informationat positions in front of the other subject regions and all commentregions. Alternatively, it is good to display the comments provided forthe subject regions at positions in front of the other subject regionsand all comment regions. By doing so, the user can easily select thesubject region as the comment input target while viewing the othercomments provided for the comment input target. As a matter of course,the information stored in the user ID information server may be modifiedin real time to reflect each user activity on the Internet.

Alternatively, with reference to the information stored in the commentinformation storage server, the input supporting apparatus may determinethe target subject region with a large number of comments to be a regionto which a user is highly likely to input a comment, for example, beforethe user inputs his or her comment.

In addition, the user may intend to input his or her comment to a givencomment A. In this case, the visibility of the comment A is moreimportant than that of the subject region to which the comment A isprovided. Accordingly, the input supporting apparatus may display thecomment region including the comment A and the subject regioncorresponding to the comment A at foremost positions. This allows theuser to easily select the comment input target region while viewing thecomment A, and to input his or her comment.

This can be performed by estimating the comment to which the userintends to input his or her comment. As described in connection with theearlier-provided example, when user ID and information such as user'shuman relationship (a social graph) and a follow state on an SNS arestored in an associated manner in the user ID information server, orwhen it is possible to access the user's social graph associated withthe user ID by the input supporting apparatus, comments provided byuser's friends and other users followed by the user may be determined tobe the comments to which the user provides his or her comment.

Alternatively, it is good to employ comments for subjects stored in theuser ID information server as matching the user's tastes. In this way,as shown in the example in (c) of FIG. 30B, it is possible to preventocclusion of the particular subject region or the comment region towhich the user tries to input his or her comment.

As for which one of the comment region and the subject region should beprioritized in the display, the user may register the one in the user IDinformation server in advance. When only one of the comment region andthe subject region is present (or can be estimated) in a frame amongsubject regions and comment regions to which the user is likely toprovide his or her comment, the only one may be prioritized.

Embodiment 3

The comment-provided video generating apparatuses in Embodiments 1 and 2each receive the video from the video server, as an input to the dataobtaining unit, and processes the video. However, instead of receivingthe video as the input from the video server, a user who uses acomment-provided video generating apparatus may input a video capturedby a user's camera to the comment-provided video generating apparatus.

FIG. 31 shows a structure of the comment-provided video generatingapparatus 1400 according to Embodiment 3. The comment-provided videogenerating apparatus 1400 includes a subject region informationgenerating unit 1402, a comment superimposition information generatingunit 1401, an image composing unit 102, and an image output unit 103.The comment-provided video generating apparatus 1400 according toEmbodiment 3 is an apparatus which generates a comment-provided video tobe displayed on a mobile terminal including the comment-provided videogenerating apparatus 1400. The mobile terminal includes a camera 1410, adisplay 120, a GPS 1411 a, and a compass 1411 b. The camera 1410captures a space in front of a user (an image capturing person). Thedisplay 120 is set at a position with an angle that allows the user toview the comment-provided video generated by the comment-provided videogenerating apparatus 1400, and displays the video. The GPS 1411 ameasures position information of the mobile terminal. The compass 1411 bmeasures a direction of the mobile terminal. The comment-provided videogenerating apparatus 1400 composes the video captured by the camera 1410and comment information obtained from an external database through acommunication path such as a network so as to generate a video in whicha comment region including the comment information is superimposed.

The subject region information generating unit 1402 obtains the videofrom the camera, and calculates subject region information. The commentsuperimposition information generating unit 1401 includes a dataobtaining unit 1404, a comment position estimating unit 1403, an inputreceiving unit 105, a display priority calculating unit 106, and acomment superimposing method determining unit 107.

The data obtaining unit 1404 obtains comment information 1412 includingthe GPS information of other terminals located within a certain rangefrom the user and supplemental information thereof. The comment positionestimating unit 1403 calculates a position of the camera in the user'smobile terminal (hereinafter referred to as a first mobile terminal) anda direction of a camera axis based on the GPS information of the firstmobile terminal and mobile terminals (hereinafter referred to as secondmobile terminals) of the other persons and compass information of thefirst mobile terminal, and estimates, for each comment information 1412,which one of subject regions in the video corresponds to the commentinformation 1412 based on the result of the calculation. The displaypriority calculating unit 106 receives, as inputs, the subject regioninformation calculated by the subject region information generating unit1402, the comment information 1412, the user information 113, and theresult of the estimation by the comment position estimating unit 1403,and calculates a display position for the comment, and display priorityfor each of the subject region and the comment. The commentsuperimposing method determining unit 107 determines a method forsuperimposing the comment onto the video, based on the display prioritycalculated by the display priority calculating unit 106.

The image composing unit 102 superimposes the comment region onto thevideo, based on the comment superimposing method determined by thecomment superimposing method determining unit 107. The image output unit103 outputs the video including the superimposed comment to the imageoutput unit 103.

FIG. 32 shows a simple configuration example of the video distributingsystem including the comment-provided video generating apparatus in thisembodiment. The video distributing system includes a comment informationstorage server and a user ID information server. Furthermore, thecomment-provided video generating apparatus 1400 may access externalserver such as an electric commerce (EC) site and information thereinthrough a network. These servers may be managed in the same PC, or inseparate PCs.

The user can register user information in the user ID information serverin advance. For example, the user can register information items of theuser in the user ID information server using a keyboard or softwarekeyboard of the mobile terminal, a touch operation, or the like.Examples of such information items include photographs or videos ofgoods bought by the user, product information or comments associatedwith the goods, link information to the EC site, information of goods onsale in the EC site and comments associated with the goods on sale. Forexample, a user A can register information items such as “one piece” asa product category, “Today's coordination” as a comment, and “2012.5.5”as a date and time, More specifically, it is assumed that the user Aregisters goods to be candidate garments and belongings of the user onthe date. Here, the date and time at which the comment is registered maybe automatically registered. When the user plans to wear garments, theregistered garments or goods (including garments and belongings), theuser may input the dates and time of wearing the garments or bringingthe belongings.

The information including these comments (comment information 1412) isadditionally stored in the comment information storage server as shownin FIG. 32. Examples of the comment information items include the userID of the user who made the comments, dates and time, photographs,categories of the goods, and information of the EC site (the name of thesite, product numbers, etc).

Here, it is assumed that other users B and C direct their cameras(activated cameras) of their mobile terminals after the user A made thecomments. Among the comment information items 1412 associated with theuser A, comment information items corresponding to today are distributedfrom the comment information storage server. These comment informationitems of the user A are superimposed and displayed on the displayscreens of the mobile terminals of the users B and C as shown in FIG.32. Alternatively, the users B and C may direct their cameras to theuser A at totally different places on different time zones of the sameday. Here, the information items related to the user A may be displayedas shown in FIG. 32 while the cameras are capturing the user A in thedisplay screens. As described earlier, the user A registers the today'sgarments or belongs in advance. Thus, for example, information relatedto the one piece worn by the user A is displayed on the display screensof the mobile terminals of the users B and C. In this way, the users Band C can know the information (the manufacturer, price, availability,etc.) about the one piece.

Conventionally, the garments etc. worn by “another person (the user Ahere)” can be known only by directly asking the user A about thegarments when other users (the users B and C here) want to know theinformation. With this configuration, the users B and C can easily knowthe information of the garments etc. of the user A.

For example, as in FIG. 32, the comment information may be displayed ata predetermined position. When the comment information items of the userA are associated with information items on the EC site, provision oflink function to images, comment characters etc. on the EC site allows auser to transit to product pages (for purchase) on the EC site only bytouching the images, comment characters etc. as a user input. Thisprovides an effect that the users who are interested in garments arepromoted to take purchase actions.

As shown in FIG. 32, when the display screens of the mobile terminals ofthe users B and C display the same images of the users A and D, and theuser D also registered in advance the same information as theinformation of the user A, one of the user A and the user D and thecomment information associated with the one are occluded by each other.Here, by controlling how to superimpose and display the subject regionsand the comment regions of the users A and D, the user B or C can knowparticular information smoothly. In view of this, Embodiment 3 performssuch control, by applying the comment-provided video generatingapparatus described in Embodiment 2. By doing so, for example, the userB who selected the user A can visually recognize the user A and thecomment information corresponding to the user A which are not occluded.Likewise, the user C who selected the user D can visually recognize theuser D and the comment information corresponding to the user D which arenot occluded.

Hereinafter, a detailed description is given of the structure of thecomment-provided video generating apparatus 1400 according to thisembodiment.

As shown in FIG. 33, the comment-provided video generating apparatus1400 according to this embodiment is especially intended to allow agiven user who is directing the camera of his or her mobile terminaltoward a subject such as a person to thereby visually recognize, on thedisplay screen of the mobile terminal, information of the person orinformation of goods belonging to the person. The comment-provided videogenerating apparatus 1400 adds and displays the information as commentfor a particular subject region in a video captured by the camera of themobile terminal and on the display screen of the mobile terminal.

The particular subject region is a subject region specified by the userthrough a user interface (UI) such as a touch panel on the mobileterminal or a subject region determined based on information registeredin an external database obtained by the data obtaining unit 1404. Thecomment-provided video generating apparatus 1400 superimposes thecomment region at a position in the video at which the comment regiondoes not occlude the particular subject region.

The subject region information generating unit 1402 is similar to thesubject region information generating unit 903 in Embodiment 2, and thusno description is given here.

FIG. 34 shows a scene in which a user specifies a subject 1. The inputreceiving unit 105 receives information input by the user using a touchpanel or the like, and calculates a user-specified position in an imagecoordinate system D.

Here, a straight line L (a straight line passing through an arbitrarypoint (xd, yd) and the origin of the camera coordinate system C) in theworld coordinate system W specified by the point (xd, yd) on an imagesuch as the user-specified position can be calculated from parameters ofthe camera and a transform matrix of the camera coordinate system. Theparameters and transform matrix can be calculated by calibration. Thisis specifically described in Non-patent Literature 1, and thus is notdescribed here.

On the other hand, an x component and a z component of positioninformation (xw, yw, zw) on the world coordinate W of each of subjectscan be obtained, for example, using GPS information from the mobileterminal of the subject. As shown in FIG. 34, a y component is acomponent in the height direction of the subject. For example, when theuser registers his height H, or an average height H is calculated basedon generation and sex, the y component is determined within a range(yw=0 to H) for use. Likewise, xw and/or zw may be determined within arange based on a value indicating an average body width.

When the earlier-mentioned straight line L is calculated, the subjectcorresponding to (xw, yw, zw) closest to the user on the path of the Lis determined to be the user-specified subject.

Based on the coordinates corresponding to each of the subject regions onan image coordinate system D calculated by the subject regioninformation generating unit 1402, the comment position estimating unit1403 estimates the mobile terminal corresponding to the subject regionof the subject using in the same manner the camera parameters andtransform matrix from GPS information of the mobile terminal locatednearby obtained by the data obtaining unit 1404. The comment positionestimating unit 1403 associates supplemental information correspondingto the mobile terminal and the subject region, and generates commentdisplay information such as a comment position, a comment shape, andcharacter information to be displayed in the video.

The display priority calculating unit 106 calculates display prioritiesfrom the comment display information generated by the comment positionestimating unit 1403, the subject region information calculated by thesubject region information generating unit 1402, and the userinformation 113 received by the input receiving unit 105. Operationsperformed by the display priority calculating unit 106 and subsequentoperations are the same as in Embodiments 1 and 2, and thus are notdescribed here.

In this way, the comment-provided video generating apparatus 1400 inthis embodiment generates a comment region which does not affect thevisibility of the user-specified subject region, and displays thesupplemental information corresponding to the subject region on thedisplay screen as shown in FIG. 33.

It is to be noted that the display priority calculating unit 106 doesnot always need to use the user information 113 received by the inputreceiving unit 105. For example, the comment information storage servermay manage popularity and the like of a particular product or categoryetc. by additionally counting the number of comments submitted for theparticular product or category with reference to data stored in thecomment information storage server. The display priority calculatingunit 106 may perform weighting on the display priorities, based on theinformation managed by the comment information storage server.

In addition, as described earlier with reference to FIG. 32, the commentinformation stored in the comment information storage server may includelink information to the EC site, product numbers of goods etc. In thiscase, when a user touches a comment or an image with the linkinformation on a comment region displayed on the mobile terminal of theuser, a product purchase page or the like automatically appears(displayed web pages are switched). In other words, the link informationmay be displayed in the comment region (balloon) independently ortogether with the comments associated with the mobile terminal. Forexample, by displaying link information to a selling site providinginformation of garments of a subject, a user can easily buy favoritegarments through the EC site.

According to Embodiment 3, it is possible to generate and display acomment-provided video in which a comment region corresponding to auser-specified subject region is not occluded by another comment regionwhen a comment corresponding to a subject in a video captured by theuser is displayed as in the case of using an Augmented Reality (AR)technique.

Each of the structural elements in each of the above-describedembodiments may be configured in the form of an exclusive hardwareproduct, or may be realized by executing a software program suitable forthe structural element. Each of the structural elements may be realizedby means of a program executing unit, such as a CPU and a processor,reading and executing the software program recorded on a recordingmedium such as a hard disk or a semiconductor memory. Here, the softwareprogram for realizing the comment-provided video generating apparatusaccording to each of the embodiments is a program described below.

The program causes a computer to execute: obtaining a video from a videoserver, and obtain, from a comment information storage server, commentinformation including (i) comments regarding the video, and (ii) displaytime and display positions of comment regions including the comments inthe video; calculating priorities for comments corresponding to thevideo and subject regions which are regions of subjects included in thevideo, based on the comment information and the user information;determining a display order or a comment superimposing method includinga transparency for superimposing and displaying the comments and thesubject regions, based on the display priorities calculated in thecalculating; generating a comment-provided video by composing thecomments on the video, based on the comment superimposing methoddetermined in the determining; and outputting the comment-provided videogenerated in the combining.

It is to be noted that the comment-provided video generating method andprogram described in the above embodiments can be effectively used as acomment superimposition order information generating method and programfor generating, as information, an order of superimposing and displayingone or more user-selected subject regions with comments, one or morecomment regions provided to the subject regions, one or moreuser-selected comment regions, and/or one or more subject regions towhich comments are provided, above the other subject regions and theother comment regions provided to the subject regions.

In addition, the comment-provided video generating method and programdescribed in the above embodiments can be effectively used as acomment-provided video output method and program for outputting agenerated comment-provided video.

Although the comment-provided video generating apparatuses according tothe aspects have been described based on the above embodiments, thoseskilled in the art will readily appreciate that these embodiments arenon-limiting examples, and various modifications may be made in theseexemplary embodiments without materially departing from the principlesand spirit of the inventive concept, the scope of which is defined inthe appended Claims and their equivalents.

The herein disclosed subject matter is to be considered descriptive andillustrative only, and the appended Claims are of a scope intended tocover and encompass not only the particular embodiments disclosed, butalso equivalent structures, methods, and/or uses.

INDUSTRIAL APPLICABILITY

The comment-provided video generating apparatuses etc. according to theexemplary embodiments disclosed herein and variations thereof areapplicable as comment-provided video generating apparatus etc. for usein audio visual apparatuses such as video cameras, television receivers,mobile terminals, tablet PCs etc. all connectable to networks and withwhich video distribution services and/or SNS on networks are available.

The invention claimed is:
 1. A comment-provided video generatingapparatus comprising: a processor; and a non-transitory memory havingstored thereon executable instructions, which when executed, cause theprocessor to perform: obtaining a video from a video server, andobtaining, from a comment information storage server, commentinformation including (i) comments regarding the video, and (ii) displaytime and display positions of comment regions including the comments inthe video; receiving, as a user input, user information including aspecified time and a specified position in the video; calculating adisplay priority for each of the comment regions and subject regions towhich the comments are provided, based on the user information andassociation information indicating associations between the comments andthe subject regions, so that the display priority calculated for (i) afirst comment region and a first subject region which correspond to thespecified time and the specified position in the video included in theuser information received in the receiving and (ii) either (a) a secondsubject region which is provided with a comment included in the firstcomment region which corresponds to the specified time and the specifiedposition in the video and included in the user information received inthe receiving or (b) a second comment region which includes a commentprovided to the first subject region which corresponds to the specifiedtime and the specified position in the video and included in the userinformation received in the receiving is higher than the displaypriority calculated for each of other comment regions and other subjectregions; determining a superimposing method for superimposing thecomment regions and the subject regions according to the displaypriority calculated for each of the comment regions and the subjectregions in the calculating such that a comment region or a subjectregion which has a higher display priority is positioned more frontwardthan a position of a comment region or a subject region which has alower display priority; generating a comment-provided video by composingthe comment regions in the video, according to the superimposing methoddetermined in the determining; and outputting the comment-provided videogenerated in the generating, wherein the outputting includes outputting,onto an image display, the comment-provided video in which either (i) afirst pair of the first subject region and the second comment regionwhich includes the comment provided to the first subject region or (ii)a second pair of the first comment region and the second subject regionwhich is provided with the comment included in the first comment regionis superimposed and displayed on the other comment regions and the othersubject regions, the first subject region and the first comment regioncorresponding to a user-selected region located at the specifiedposition indicated by the user information, and wherein the determiningdetermines the superimposing method such that only a comment regionwhich is displayed at a foremost position in each of frames of the videoin a case of normal reproduction among the comment regions in the frameis displayed during a period in which a viewing method which is otherthan the normal reproduction and includes fast forward, preview, rewind,and skip is performed on the comment-provided video.
 2. Thecomment-provided video generating apparatus according to claim 1,wherein the display priority calculated for the first comment region andthe first subject region is higher than the display priority calculatedfor either the second subject region and the second comment region. 3.The comment-provided video generating apparatus according to claim 1,wherein the determining determines, based on depth relationship betweenthe subject regions in the video, the superimposing method such thatcomment regions other than the first comment region and the secondcomment region are positioned based on a depth relationship between thesubject regions in the video.
 4. The comment-provided video generatingapparatus according to claim 3, wherein the calculating calculates ahigher display priority for a comment region including a comment whichis provided later in either the display time included in the commentinformation or in real submission time of the comments.
 5. Thecomment-provided video generating apparatus according to claim 1,wherein the executable instructions further cause the processor toperform obtaining the user information from a user ID informationserver, and wherein, when user ID information in the user ID informationserver is updated while the user is viewing the video, the calculatingincludes calculating a new display priority for each of the commentregions and the subject regions, based on the updated user IDinformation.
 6. The comment-provided video generating apparatusaccording to claim 1, wherein the determining includes calculating, foreach of pixels in an overlap area between two regions which are either(i) one of the subject regions and one of the comment regions or (ii)two of the comment regions, a transparency of the pixel to be displayedsemi-transparent, based on a display priority ratio of the displaypriorities calculated for the two regions including the pixels, andwherein the determining determines the superimposing method such thatone of the two regions which is superimposed at the position morefrontward than the position of the other one of the two regions isdisplayed more opaquely according to the calculated transparency.
 7. Thecomment-provided video generating apparatus according to claim 1,wherein the determining determines, based on information describing anocclusion relationship between subject regions among the subjectregions, the superimposing method such that comment regions includingcomments corresponding to the subject regions in the occlusionrelationship are displayed by temporally stopping the comment regions atpositions before a start of an occlusion period in which the subjectregions are in the occlusion relationship.
 8. The comment-provided videogenerating apparatus according to claim 1, wherein the determiningdetermines, based on information describing an occlusion relationshipbetween subject regions, the superimposing method such that commentregions including comments provided to the subject regions in theocclusion relationship are displayed by integrating the comment regionsat positions before a start of an occlusion period in which the subjectregions are in the occlusion relationship into a balloon comment region.9. The comment-provided video generating apparatus according to claim 1,wherein the determining determines, based on information describing anocclusion relationship between subject regions among the subjectregions, the superimposing method such that only a comment regioncorresponding to an occluding subject region closer to a location of acamera used to capture the video among comment regions includingcomments provided to the subject regions in the occlusion relationshipis displayed during an occlusion period in which the subject regions arein the occlusion relationship.
 10. The comment-provided video generatingapparatus according to claim 9, wherein the determining determines thesuperimposing method and a transparency for superimposing the commentregions such that one of the comment regions which includes the commentprovided to the occluding subject region is displayed semi-transparent,and all of the comment regions are displayed non-semi-transparent afterthe occlusion period.
 11. A comment-provided video generating apparatuscomprising: a processor; and a non-transitory memory having storedthereon executable instructions, which when executed, cause theprocessor to perform: obtaining a video from a video server, andobtaining, from a comment information storage server, commentinformation including (i) comments regarding the video, and (ii) displaytime and display positions of comment regions including the comments inthe video; receiving, as a user input, user information including aspecified time and a specified position in the video; calculating adisplay priority for each of the comment regions and subject regions towhich the comments are provided, based on the user information andassociation information indicating associations between the comments andthe subject regions, so that the display priority calculated for (i) afirst comment region and a first subject region which correspond to thespecified time and the specified position in the video included in theuser information received in the receiving and (ii) either (a) a secondsubject region which is provided with a comment included in the firstcomment region which corresponds to the specified time and the specifiedposition in the video and included in the user information received inthe receiving or (b) a second comment region which includes a commentprovided to the first subject region which corresponds to the specifiedtime and the specified position in the video and included in the userinformation received in the receiving is higher than the displaypriority calculated for each of other comment regions and other subjectregions; determining a superimposing method for superimposing thecomment regions and the subject regions according to the displaypriority calculated for each of the comment regions and the subjectregions in the calculating such that a comment region or a subjectregion which has a higher display priority is positioned more frontwardthan a position of a comment region or a subject region which has alower display priority; generating a comment-provided video by composingthe comment regions in the video, according to the superimposing methoddetermined in the determining; outputting the comment-provided videogenerated in the generating; and obtaining the user information from auser ID information server, wherein the outputting includes outputting,onto an image display, the comment-provided video in which either (i) afirst pair of the first subject region and the second comment regionwhich includes the comment provided to the first subject region or (ii)a second pair of the first comment region and the second subject regionwhich is provided with the comment included in the first comment regionis superimposed and displayed on the other comment regions and the othersubject regions, the first subject region and the first comment regioncorresponding to a user-selected region located at the specifiedposition indicated by the user information, when user ID information inthe user ID information server is updated while the user is viewing thecomment-provided video, the calculating includes calculating a newdisplay priority for each of the comment regions and the subjectregions, based on the updated user ID information, the obtaining theuser information includes obtaining viewing history of the user storedexternally or in the user ID information server, when thecomment-provided video which has been viewed by a user is viewed by theuser, the calculating includes providing a comment region or a subjectregion displayed at a foremost position when viewed last time by theuser with a lowest display priority, and in the comment-provided videodisplayed on the image display, the comment region displayed at theforemost position when viewed last time by the user is displayed at arearmost position.
 12. A comment superimposing method determining methodcomprising: obtaining a video from a video server, and obtaining, from acomment information storage server, comment information including (i)comments regarding the video, and (ii) display time and displaypositions of comment regions including the comments in the video;receiving, as a user input, user information including a specified timeand a specified position in the video; calculating a display priorityfor each of the comment regions and subject regions to which thecomments are provided, based on the user information and associationinformation indicating associations between the comments and the subjectregions, so that the display priority calculated for (i) a first commentregion and a first subject region which correspond to the specified timeand the specified position in the video included in the user informationreceived in the receiving and (ii) either (a) a second subject regionwhich is provided with a comment included in the first comment regionwhich corresponds to the specified time and the specified position inthe video and included in the user information received in the receivingor (b) a second comment region which includes a comment provided to thefirst subject region which corresponds to the specified time and thespecified position in the video and included in the user informationreceived in the receiving is higher than the display priority calculatedfor each of other comment regions and other subject regions; determininga superimposing method for superimposing the comment regions and thesubject regions according to the display priority calculated for each ofthe comment regions and the subject regions in the calculating such thata comment region or a subject region which has a higher display priorityis positioned more frontward than a position of a comment region or asubject region which has a lower display priority, wherein thedetermining determines the superimposing method such that only a commentregion which is displayed at a foremost position in each of frames ofthe video in a case of normal reproduction among the comment regions inthe frame is displayed during a period in which a viewing method whichis other than the normal reproduction and includes fast forward,preview, rewind, and skip is performed on the comment-provided video.13. A comment-provided video generating method comprising: Thegenerating superimposing method determining method according to claim12; generating the comment-provided video by composing the commentregions in the video according to the comment superimposing methoddetermined in the determining; and outputting the comment-provided videogenerated in the generating, wherein, in the comment-provided videooutput in the outputting, either (i) a first pair of the first subjectregion and the second comment region which includes the comment providedto the first subject region or (ii) a second pair of the first commentregion and the second subject region which is provided with the commentincluded in the first comment region is superimposed and displayed onthe other comment regions and the other subject regions, the firstsubject region and the first comment region corresponding to auser-selected region located at the specified position indicated by theuser information.
 14. The comment-provided video generating methodaccording to claim 13, wherein the association information furtherindicates subject information of subjects to which the plurality ofcomments are provided.
 15. A comment-provided video generating apparatuscomprising: a processor; and a non-transitory memory having storedthereon executable instructions, which when executed, cause theprocessor to perform: obtaining a video captured by a camera from amobile terminal, the mobile terminal including (i) the camera which isused to capture an image of a user positioned in front of the camera,(ii) a display which is provided at a position that allows the user tovisually recognize the display, (iii) a Global Positioning System (GPS),and (iv) a compass; generating subject region information indicating aplurality of subject regions in a frame of the obtained video based oncontent of the video, the subject regions being generated in the frameby dividing the frame; obtaining comment information including (i) GPSinformation of other mobile terminals located in a predetermined rangefrom the user and (ii) a plurality of comments associated one to one tothe other mobile terminals; receiving, as a user input, user informationincluding a specified position in the video; calculating positions ofthe other mobile terminals in the video, based on the GPS information ofthe mobile terminal of the user, the GPS information of the other mobileterminals, and information of the compass of the mobile terminal;associating one to one subject regions presenting subjects who are usersof the other mobile terminals and the comments included in the commentinformation obtained in the obtaining the comment information, based onthe calculated positions of the other mobile terminals; calculating,based on the comment information and the user information, a displaypriority for each of a plurality of comment regions which aresuperimposed and displayed in the video and the subject regions in thevideo by calculating, for a first comment region including a firstcomment corresponding to a first subject region located at a displayposition in the video specified by the user information received in thereceiving among the comment regions, a display priority which is higherthan the display priority calculated for the for each of the othersubject regions and the other comment regions; determining, as asuperimposing method for displaying the comment regions in the video,one of a first method and a second method according the display prioritycalculated in the calculating; the first method being for determining adisplay order for displaying the comment regions and the subject regionsin the video on an image display by superimposing the comment regionsand the subject regions according to the display priority calculated foreach of the comment regions and the subject regions in the calculatingsuch that a comment region or a subject region which has a higherdisplay priority is positioned more frontward than a position of acomment region or a subject region which has a lower display priority,and the second method being for determining a transparency fordisplaying the comment regions and the subject regions according to thedisplay priority calculated for each of the comment regions and thesubject regions in the calculating such that a comment region or asubject region which has a higher transparency is displayed moreopaquely; superimposing the comment regions in the video, according tothe comment superimposing method determined in the determining; andoutputting the comment-provided video in which the comment regions aresuperimposed in the superimposing, wherein the determining determinesthe superimposing method such that only a comment region which isdisplayed at a foremost position in each of frames of the video in acase of normal reproduction among the comment regions in the frame isdisplayed during a period in which a viewing method which is other thanthe normal reproduction and includes fast forward, preview, rewind, andskip is performed on the comment-provided video.
 16. A comment-providedvideo generating method comprising: obtaining a video from a videoserver; obtaining, from a comment information storage server, commentinformation including (i) comments regarding the video, and (ii) displaytime and display positions of comment regions including the comments inthe video; receiving, as a user input, user information including aspecified time and a specified position; and generating acomment-provided video, based on the user information and associationinformation indicating one-to-one associations between the comments andthe subject regions, by superimposing and displaying (i) either a firstcomment region or a first subject region which correspond to thespecified time and the specified position on the video included in theuser information received in the receiving, at a foremost position inthe video, and (ii) either (a) a second subject region which is providedwith a comment included in the first comment region which corresponds tothe specified time and the specified position on the video and includedin the user information received in the receiving or (b) a secondcomment region which includes a comment provided to the first subjectregion which corresponds to the specified time and the specifiedposition on the video and included in the user information received inthe receiving, at a position next frontward to the foremost position inthe video, wherein the generating includes determining a superimposingmethod for superimposing comment regions and subject regions of thevideo such that only a comment region which is displayed at a foremostposition in each of frames of the video in a case of normal reproductionamong the comment regions in the frame is displayed during a period inwhich a viewing method which is other than the normal reproduction andincludes fast forward, preview, rewind, and skip is performed on thecomment-provided video.
 17. A comment-provided video generatingapparatus comprising: a processor; and a non-transitory memory havingstored thereon executable instructions, which when executed, cause theprocessor to perform: obtaining a video from a video server, andobtaining, from a comment information storage server, commentinformation including (i) comments regarding the video, and (ii) displaytimes and display positions of comment regions including the comments inthe video; receiving, as a user input, user information including aspecified time and a specified position; calculating a display priorityfor each of the comment regions and subject regions to which commentsincluded in the comment regions are provided, based on the specifiedtime and the specified position included the user information receivedin the receiving and association information indicating one-to-oneassociations between the comments and the subject regions; andoutputting a comment-provided video according to the display prioritycalculated for each of the comment regions and the subject regions inthe calculating such that a comment region or a subject region which hasa higher display priority is superimposed and displayed at a positionmore frontward than a position of a comment region or a subject regionwhich has a lower display priority, wherein the outputting includesdetermining a superimposing method for superimposing comment regions andsubject regions of the video such that only a comment region which isdisplayed at a foremost position in each of frames of the video in acase of normal reproduction among the comment regions in the frame isdisplayed during a period in which a viewing method which is other thanthe normal reproduction and includes fast forward, preview, rewind, andskip is performed on the comment-provided video.
 18. A commentsuperimposing method determining method comprising: obtaining a videofrom a video server, and obtaining, from a comment information storageserver, comment information including (i) comments regarding the video,and (ii) display time and display positions of comment regions includingthe comments in the video; receiving, as a user input, user informationincluding a specified time and a specified position in the video;calculating a display priority for each of the comment regions andsubject regions to which the comments are provided, based on the userinformation and association information indicating associations betweenthe comments and the subject regions, so that the display prioritycalculated for (i) a first comment region and a first subject regionwhich correspond to the specified time and the specified position in thevideo included in the user information received in the receiving and(ii) either (a) a second subject region which is provided with a commentincluded in the first comment region which corresponds to the specifiedtime and the specified position in the video and included in the userinformation received in the receiving or (b) a second comment regionwhich includes a comment provided to the first subject region whichcorresponds to the specified time and the specified position in thevideo and included in the user information received in the receiving ishigher than the display priority calculated for each of other commentregions and other subject regions; determining a superimposing methodfor superimposing the comment regions and the subject regions accordingto the display priority calculated for each of the comment regions andthe subject regions in the calculating such that a comment region or asubject region which has a higher display priority is positioned morefrontward than a position of a comment region or a subject region whichhas a lower display priority; generating a comment-provided video bycomposing the comment regions in the video, according to thesuperimposing method determined in the determining; outputting thecomment-provided video generated in the generating; and obtaining theuser information from a user ID information server, wherein theoutputting includes outputting, onto an image display, thecomment-provided video in which either (i) a first pair of the firstsubject region and the second comment region which includes the commentprovided to the first subject region or (ii) a second pair of the firstcomment region and the second subject region which is provided with thecomment included in the first comment region is superimposed anddisplayed on the other comment regions and the other subject regions,the first subject region and the first comment region corresponding to auser-selected region located at the specified position indicated by theuser information, when user ID information in the user ID informationserver is updated while the user is viewing the comment-provided video,the calculating includes calculating a new display priority for each ofthe comment regions and the subject regions, based on the updated userID information, the obtaining the user information includes obtainingviewing history of the user stored externally or in the user IDinformation server, when the comment-provided video which has beenviewed by a user is viewed by the user, the calculating includesproviding a comment region or a subject region displayed at a foremostposition when viewed last time by the user with a lowest displaypriority, and in the comment-provided video displayed on the imagedisplay, the comment region displayed at the foremost position whenviewed last time by the user is displayed at a rearmost position.
 19. Acomment-provided video generating apparatus comprising: a processor; anda non-transitory memory having stored thereon executable instructions,which when executed, cause the processor to perform: obtaining a videocaptured by a camera from a mobile terminal, the mobile terminalincluding (i) the camera which is used to capture an image of a userpositioned in front of the camera, (ii) a display which is provided at aposition that allows the user to visually recognize the display, (iii) aGlobal Positioning System (GPS), and (iv) a compass; generating subjectregion information indicating a plurality of subject regions in a frameof the obtained video based on content of the video, the subject regionsbeing generated in the frame by dividing the frame; obtaining commentinformation including (i) GPS information of other mobile terminalslocated in a predetermined range from the user and (ii) a plurality ofcomments associated one to one to the other mobile terminals; receiving,as a user input, user information including a specified position in thevideo; calculating positions of the other mobile terminals in the video,based on the GPS information of the mobile terminal of the user, the GPSinformation of the other mobile terminals, and information of thecompass of the mobile terminal; associating one to one subject regionspresenting subjects who are users of the other mobile terminals and thecomments included in the comment information obtained in the obtainingthe comment information, based on the calculated positions of the othermobile terminals; calculating, based on the comment information and theuser information, a display priority for each of a plurality of commentregions which are superimposed and displayed in the video and thesubject regions in the video by calculating, for a first comment regionincluding a first comment corresponding to a first subject regionlocated at a display position in the video specified by the userinformation received in the receiving among the comment regions, adisplay priority which is higher than the display priority calculatedfor the for each of the other subject regions and the other commentregions; determining, as a superimposing method for displaying thecomment regions in the video, one of a first method and a second methodaccording the display priority calculated in the calculating; the firstmethod being for determining a display order for displaying the commentregions and the subject regions in the video on an image display bysuperimposing the comment regions and the subject regions according tothe display priority calculated for each of the comment regions and thesubject regions in the calculating such that a comment region or asubject region which has a higher display priority is positioned morefrontward than a position of a comment region or a subject region whichhas a lower display priority, and the second method being fordetermining a transparency for displaying the comment regions and thesubject regions according to the display priority calculated for each ofthe comment regions and the subject regions in the calculating such thata comment region or a subject region which has a higher transparency isdisplayed more opaquely; superimposing the comment regions in the video,according to the comment superimposing method determined in thedetermining; outputting the comment-provided video in which the commentregions are superimposed in the superimposing; and obtaining the userinformation from a user ID information server, wherein when user IDinformation in the user ID information server is updated while the useris viewing the comment-provided video, the calculating includescalculating a new display priority for each of the comment regions andthe subject regions, based on the updated user ID information, theobtaining the user information includes obtaining viewing history of theuser stored externally or in the user ID information server, when thecomment-provided video which has been viewed by a user is viewed by theuser, the calculating includes providing a comment region or a subjectregion displayed at a foremost position when viewed last time by theuser with a lowest display priority, and in the comment-provided videodisplayed on the image display, the comment region displayed at theforemost position when viewed last time by the user is displayed at arearmost position.
 20. A comment-provided video generating methodcomprising: obtaining a video from a video server; obtaining, from acomment information storage server, comment information including (i)comments regarding the video, and (ii) display time and displaypositions of comment regions including the comments in the video;receiving, as a user input, user information including a specified timeand a specified position; generating a comment-provided video, based onthe user information and association information indicating one-to-oneassociations between the comments and the subject regions, bysuperimposing and displaying (i) either a first comment region or afirst subject region which correspond to the specified time and thespecified position on the video included in the user informationreceived in the receiving, at a foremost position in the video, and (ii)either (a) a second subject region which is provided with a commentincluded in the first comment region which corresponds to the specifiedtime and the specified position on the video and included in the userinformation received in the receiving or (b) a second comment regionwhich includes a comment provided to the first subject region whichcorresponds to the specified time and the specified position on thevideo and included in the user information received in the receiving, ata position next frontward to the foremost position in the video; andobtaining the user information from a user ID information server,wherein when user ID information in the user ID information server isupdated while the user is viewing the video, the calculating includescalculating a new display priority for each of the comment regions andthe subject regions, based on the updated user ID information, theobtaining the user information includes obtaining viewing history of theuser stored externally or in the user ID information server, when thecomment-provided video which has been viewed by a user is viewed by theuser, the calculating includes providing a comment region or a subjectregion displayed at a foremost position when viewed last time by theuser with a lowest display priority, and in the comment-provided videodisplayed on an image display, the comment region displayed at theforemost position when viewed last time by the user is displayed at arearmost position.
 21. A comment-provided video generating apparatuscomprising: a processor; and a non-transitory memory having storedthereon executable instructions, which when executed, cause theprocessor to perform: obtaining a video from a video server, andobtaining, from a comment information storage server, commentinformation including (i) comments regarding the video, and (ii) displaytimes and display positions of comment regions including the comments inthe video; receiving, as a user input, user information including aspecified time and a specified position; calculating a display priorityfor each of the comment regions and subject regions to which commentsincluded in the comment regions are provided, based on the specifiedtime and the specified position included the user information receivedin the receiving and association information indicating one-to-oneassociations between the comments and the subject regions; outputting acomment-provided video according to the display priority calculated foreach of the comment regions and the subject regions in the calculatingsuch that a comment region or a subject region which has a higherdisplay priority is superimposed and displayed at a position morefrontward than a position of a comment region or a subject region whichhas a lower display priority; obtaining the user information from a userID information server, wherein when user ID information in the user IDinformation server is updated while the user is viewing the video, thecalculating includes calculating a new display priority for each of thecomment regions and the subject regions, based on the updated user IDinformation, the obtaining the user information includes obtainingviewing history of the user stored externally or in the user IDinformation server, when the comment-provided video which has beenviewed by a user is viewed by the user, the calculating includesproviding a comment region or a subject region displayed at a foremostposition when viewed last time by the user with a lowest displaypriority, and in the comment-provided video displayed on an imagedisplay, the comment region displayed at the foremost position whenviewed last time by the user is displayed at a rearmost position.